tag:blogger.com,1999:blog-52869569488688563112024-02-19T02:49:12.247+01:00BiodivertidoBiodiversity data harvesting, indexing, integration and user interfacesJavier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.comBlogger49125tag:blogger.com,1999:blog-5286956948868856311.post-48015731038934760862011-08-04T17:10:00.002+02:002011-08-04T17:13:10.125+02:00Announcing EcoHackNYCJust a quick note to those still following. <br /><br />We are planning a day and a half of presentations and hacking for those of you interested in global environmental change in the NYC area and elsewhere. We'll post more details as we get closer to the date, but start putting it on your calendars. It will be the first weekend of November, and you can find details here,<br /><br /><a href="http://goo.gl/uVT7E"><span style="font-weight:bold;">EcoHackNYC</span></a>Andrew W Hillhttp://www.blogger.com/profile/04814436364156670727noreply@blogger.com0tag:blogger.com,1999:blog-5286956948868856311.post-65098341855415193892010-09-06T09:09:00.000+02:002010-09-06T09:09:52.892+02:00Biodiversity and conservation agenda at FOSS4G 2010<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYfvvJ11Y6gA3SnRyoL_hmDNXLnYYh9UvGvu7imBmuiJZWLjDnTkJu14-FM0WKK4wcS9KbMAlP9C74WeLei2wDqZU3k70E0UkOPNfeAJ3XjkelqKmGu39H_ykBQZ3TnfZq-mBRw0bkjuMb/s1600/Screen+shot+2010-09-06+at+9.09.17+AM.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYfvvJ11Y6gA3SnRyoL_hmDNXLnYYh9UvGvu7imBmuiJZWLjDnTkJu14-FM0WKK4wcS9KbMAlP9C74WeLei2wDqZU3k70E0UkOPNfeAJ3XjkelqKmGu39H_ykBQZ3TnfZq-mBRw0bkjuMb/s320/Screen+shot+2010-09-06+at+9.09.17+AM.png" /></a></div>Today starts <a href="http://2010.foss4g.org/">FOSS4G 2010</a> in Barcelona the biggest event on Open Source GIS software. We have been trying to push the participation of Biodiversity related projects and tools on this event in the previous months. Unfortunately most of the talks sent by Biodiversity related projects had not been accepted. The number of abstracts sent this year to the conference is been so big, that seems it was really hard to get into the program.<br />
<br />
Nevertheless, there is still quite some Biodiversity and conservation related talks and events going on at FOSS4G and I would like to highlight from the <a href="http://2010.foss4g.org/program_print.php">official program</a> a "Biodiversity and conservation agenda at FOSS4G 2010" for those interested on the topic. There is muuuch more in the program, I am just highlighting here what I believe is relevant to our community.<br />
<b><span class="Apple-style-span" style="font-weight: normal;"><br />
</span></b><br />
<br />
<b>Tuesday 7th:</b><br />
<b>16:30</b><br />
<br />
<a href="http://2010.foss4g.org/presentations_show.php?id=2891">OpenLayers: SOS and INSPIRE</a> This is not a biodiversity specific talk, but INSPIRE will likely change any Biodiversity project done in Europe, so I pick this one, as SOS is potentially one of the most realted standards to Biodiversity.<br />
<br />
<b>Wednesday 8th</b><br />
<b>11:00</b><br />
<a href="http://2010.foss4g.org/presentations_show.php?id=3693">Enhancing the European Forest Fire Information System (EFFIS) with Open Source Software </a><br />
<br />
<b>12:00</b><br />
<a href="http://2010.foss4g.org/presentations_show.php?id=2996">Natural Earth – Free World Base Map</a><br />
<a href="http://2010.foss4g.org/presentations_show.php?id=3127">The World Meteorological Oganization Information System</a><br />
<a href="http://2010.foss4g.org/presentations_show.php?id=3721">ecoRelevé: An open source response to the biodiversity crisis </a><br />
<br />
<br />
<b>Thursday 9th:</b><br />
<b>11:00</b><br />
<a href="http://2010.foss4g.org/presentations_show.php?id=3581">Open-source Earthquake and Hydrodynamic Modelling </a><br />
<a href="http://2010.foss4g.org/presentations_show.php?id=3152">Building the Digital Observatory for Protected Areas on an Open Source Framework </a><br />
<a href="http://2010.foss4g.org/presentations_show.php?id=3743">Open Environmental Services Infrastructure </a><br />
<br />
We are also organizing a <a href="http://wiki.osgeo.org/wiki/FOSS4G_2010_Breakout_Sessions#Biodiversity.2Fconservation_projects_and_FOSS4G_tools">Break Out session for Biodiversity</a>. Please register on the wiki so that we know how much people will be coming.<br />
<br />
Finally maybe a list of people that I know are coming to FOSS4G and that are part of the biodiversity informatics community:<br />
<br />
Andrew Hill - University of Colorado<br />
Olivier Coullet - Natural solutions<br />
Javier de la Torre - Vizzuality<br />
<br />
I look forward to see you all there.Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com0tag:blogger.com,1999:blog-5286956948868856311.post-79409558926849628812010-06-07T19:49:00.027+02:002010-06-21T06:48:44.927+02:00PhyloBox: A better way to do phylogenetic trees on the webA few weeks ago Rod Page posted his experiments with <a href="http://iphylo.blogspot.com/2010/05/drawing-phylogeny-in-web-browser-using.html">displaying phylogenetic trees on the web using the HTML5 Canvas and JavaScript</a>. I was particularly interested, and a bit threatened, by this post. At the time I had been toying with the exact same idea with the hopes of entering such a solution in the <a href="http://ievobio.org/challenge.html">iEvoBio Visualization Challenge</a>. As a response, I needed to step up my game. Here are the steps for going from a simple HTML Canvas rendering a phylogenetic tree to our end result, <a href="http://phylobox.appspot.com/">PhyloBox</a>.<br /><br /><b>Step 1: Get a bigger weapon</b><br /><br />I love tinkering with code and playing with new technologies, but I am far from the top when it comes to browser based smarts. I was going to need some help on this one. My friend <a href="http://plebeian.tv/">Sander Pick</a> was nice enough to spend some time tackling the problem with me. This was particularly exciting to me, as Sander has not spent much time thinking about biology on the web, getting some fresh perspective was just what this needed (in my opinion). He dove right into thinking about how to make trees, and in particular very big-trees, accessible in the HTML5 Canvas in really cool ways. Almost every time we started talking about solutions he would bring up an idea that was entirely new and exciting for this project.<br /><br /><b>Step 2: Bring trees up to speed</b><br /><br />Phylogenetic trees are great, I love them, I have never loved their notation. Parenthetic? Might as well be punch cards. Nexus? Ok, not bad for a text-file on steroids. <a href="http://phyloxml.org">PhyloXML</a>? Here we go, this might be on to something. Yet, I’m one of those people who feel like XML is sooo 2005, or maybe the year before that. What do we need to make the data within a phylogenetic tree usuable in a web application? My vote, <a href="http://www.json.org/">JSON</a>. It is simple, fast, and perfect for applications relying on JavaScript. Although it may not be able to handle some of the more complex tree notation in PhyloXML, I haven't run into that yet.<br /><br />I went about creating a PhyloXML -> PhyloJSON converter and then Sander and I spent some time refining what exactly the data needed to look like to call it PhyloJSON and make it valuable for web-based visualization. I figured that what we do isn’t going to be the end-all of phylogenetic data on the web, so we have included the converter as a REST service for you to use (<a href="http://phylobox.appspot.com/api/convert">see here</a>). I have near-future plans to make parenthetic->PhyloJSON and Nexus/NeXML -> PhyloJSON converters also, but would like to gather some feedback first.<div><br />PhyloJSON is primarily a conversion of the elements of phyloXML (right now a limited subset, but I'm willing to expand rapidly to match demand) to a JSON object. We changed some of the element names to reduce their length and we added a bit more top level information to define the environment (e.g. where to root the tree for viewing, what color to draw the background, where to position the tree, etc). In the process, we also flatten that tree, we weren't fond of the super nested structure of PhyloXML when we wanted to reference nodes in the trees with JavaScript very quickly. You can see the spec, <a href="http://phylobox.appspot.com/phylojson/version/1">here</a>, and you can see an example of PhyloJSON <a href="http://phylobox.appspot.com/phylojson/example">here</a>.</div><div><br /><b>Step 3: Build a Web-Publishing Framework for Phylogenies</b></div><div><b><br /></b>We wanted people to not only be able to view a phylogeny on the web, we also want them to be able to edit them to their liking (while still giving an option to just view). I have been doing some development on Google App Engine anyway (see <a href="http://geophylo.appspot.com/">GeoPhylo</a>), so I was comfortable building the system there. I wanted to free myself from any hardware maintenance. But secondly, for reasons expanded below, I wanted to allow for a lot of requests at busy times (elasticity) without having to question my server admin skills. Finally, I wanted to take advantage of the free quotas, good amount of free data storage, and super-simple user management that comes with App Engine; users with existing Google accounts will automatically have accounts on our system.<br />The features of our framework are many.<br /><ul><li>Anonymous tree viewing. This allows users to use our technology without the need to actually sign-in or authenticate. So, for example, our REST services will take either a PhyloXML file or PhyloJSON file (zip supported for big ones), and return all the needed bits to view that tree on the web without any authentication (we are working on OAuth as well though for greater control including versioning and stored tree defaults). You can even use our tree editor anonymously (although there are some good bits you might like to sign-in for).</li><li>User based project management. This allows users to save and return to phylogenetic trees that they own. It also allows users to invite collaborators to edit the same tree. Or fork their tree into a new version, preserving the older ones to track progress or changes and track how many much the tree is being viewed.</li><li>Creative commons publishing. The system is open from the start. This may scare you, but let me explain. When a tree is uploaded to our site, it is assigned a UUID. If you know the UUID, you can see the tree, but can’t edit it unless you own it. That means that you can share the URL with anyone you want, but because the UUID has 16^32 combinations, no one should stumble upon it. This is a very similar method to the one that Google Docs uses if you use their ‘Get the link to share’ option. This will ensure that you can use the tree that you own for anything you darn well please. </li><li>Project forking. This is something we are excited about, and hope it doesn’t make you run the other way. If you share a link to a tree with me, but I am not an owner or collaborator on that tree, I can’t edit it. That’s too bad because I may want to show you some other branch coloring scheme that would look better, or convey the data better. In the spirit of science, we made it so that I can fork the project. When I fork it, I get a completely new UUID to a new tree object with all the same info, which I can edit and re-share with you. The history of ownership and contribution is maintained. You will always be the original author, and if anyone forks the tree that I made from yours, you are still original author. The entire ‘lineage’ of that tree is also kept so that we can reconstruct where changes came from who added what. Cool? Here is an figure of the concept.<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_R9LN1Tv4DzhHoVAv4uvyhVaThD58tqnnrY2D4AET6B9QY0gMC83sjzzCjfh5CMAld0-GH8cOK-Hgb3FQtax2gQqO7tJpCyRIBHgsActUFLqCYq22mgKIVq3Fs0S6hCr2rnuqeRvQHWmD/s1600/forking.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 400px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_R9LN1Tv4DzhHoVAv4uvyhVaThD58tqnnrY2D4AET6B9QY0gMC83sjzzCjfh5CMAld0-GH8cOK-Hgb3FQtax2gQqO7tJpCyRIBHgsActUFLqCYq22mgKIVq3Fs0S6hCr2rnuqeRvQHWmD/s400/forking.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5480862349905697442" /></a></li><li>Project management console. This aspect is far from complete, but in this area, we will give original authors ways of seeing where there idea has gone, see how often it is being viewed etc. For now, you can see a list of your trees and number of views.</li><li>A growing API, where advanced users can take advantage of each part of our system, without any use of the graphical UI. </li></ul>With the project management in place, we give people a lot of ways to take advantage of our phylogenetic tree viewing technology (which we’re getting to, relax). Let us know what you would like to see and we will think about it in all our free time ;). Just kidding, really, let us know what you would like to see.</div><br /><div><br /><b>Step 4: Build Better Web-Publishing Methods for Phylogenies</b></div><br /><div><b><br /></b>So now we have methods for you to control, create, and share your trees within our site. But what if you want the trees on your site? Yes, you could use our super simple ‘output image’ button and stick that image on your website. But don’t we all want something more than that? Not only do you want to show the tree on your site, you want to control the starting view (dendrogram or phylogram, branch lenth or not, camera starting angle) shown to users. You probably even want the tree being to be interactive. Okay, now you are getting pushy, we had to think about this one for a while. We knew that we had this Canvas element where we would render your tree, but how do you share a Canvas efficiently? And how do you make sure that all those JavaScript libraries are getting incorporated at the right times? And how can we continue to develop our services with the most up-to-date technologies next year, and still give you the best looking tree that you published on your website last year? Well, other people have already answered these questions.</div><div><br />If you look at YouTube or SlideShare, their widget always looks the same across the internet, regardless of when someone actually added it to their website. We wanted that! The problem is, we weren't going for a Flash based solution, so getting it wasn’t so straight forward. I’ll spare you a lot of the boring details on this, but we did it. After a lot of tinkering, we got a pure JavaScript widget that works in much the same way as a YouTube video. Here is an example of a PhyloBox Widget:<br /><br /><div id="PhyloboxEmbed" > <br /> <div width="375" height="344" style="width:375px;height:344px;" id="phylobox_phylobox-1-0-ecfa61d1-db2b-49a3-9c05-8f4f682e68d9" class="phylobox_embed_parent"><a href="http://phylobox.appspot.com"><img src="http://phylobox.appspot.com/api/image.png?k=phylobox-1-0-ecfa61d1-db2b-49a3-9c05-8f4f682e68d9" width="375" height="344" /></a><br /> </div> <br /></div> <br /> <br /><br />We wanted to make the code to add a widget to your page super small and easy to just plop in there. Much like the EMBED code you may be familiar with from a YouTube video, we achieved a nice small package. Here is the code it took to embed the widget above:<br /><br /><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em; max-width:300px;overflow:auto;"><br /><div id="PhyloboxEmbed" ><br /> <div width="375" height="344" style="width:375px;height:344px;" id="phylobox_phylobox-1-0-ecfa61d1-db2b-49a3-9c05-8f4f682e68d9" class="phylobox_embed_parent"><a href="http://phylobox.appspot.com"><img src="http://phylobox.appspot.com/api/image.png?k=phylobox-1-0-ecfa61d1-db2b-49a3-9c05-8f4f682e68d9" width="375" height="344" /></a><br /> </div><br /></div><br /></pre><br /><br />In fact, you can click the ‘open tag-end tag’ icon on the widget itself and grab the code to put in your website! The interaction with that tree isn’t perfect yet. We just haven’t had time to get mouse or gesture based control yet, but go ahead and play with your arrow keys, or your A or Z keys or shift+V keys. There are others, but a lot of the interaction is being developed, so we will be changing a lot of it shortly. What is also cool here, is that while the rending of the tree isn’t perfect yet, as we improve our methods, the widget on this page will always show the most up-to-date version.<br />Some small caveats. The widget works on most all personal websites, so we assume your ScratchPads page or EOL Species Page can handle it no problem (but if not, we will work to make it happen real quick). However, blogs on BlogSpot don’t really like it, in fact flat out reject it, when you stick JavaScript in your blog posts. Well, “screw that” we said. In order to get around this, we made a small Gadget for you to add to your blog, you can see it in our sidebar (just the little link to PhyloBox). It just looks for the PhyloBox Widget Div element and if it exists in the current post, pulls in the necessary JavaScript to display your tree correctly. To get it running on your blog see our <a href="http://phylobox.appspot.com/help/blogger">help page</a>.<br />What was that? A YouTube sized display is too small for you to show a tree? Yeah, good point. Well, a YouTube sized display is probably too small to show a video a lot of times too. But that is just the reality of the web, real estate is needed for all sorts of things. That is why you can control the starting size right in the small snip of code, and that is also why YouTube so kindly has that fullscreen button on every video, and so do we! How cool is that? Because we hope people plug these in all over the web, we also expect (hope for) a large number of simultaneous requests. Oh, I should mention now, we haven’t even gone as far as scratching our heads over out-dated browsers. That would be like building highways for concrete wheels, we’re just not going to do it.<br />Buuuuuuut... If you really hate all of this and still just want to be able to put together a phylogenetic tree for the web and just insert an image of it on your website, we've got you covered.<br /><br /><img style="max-width:300px; max-height:300px;" src="http://phylobox.appspot.com/api/image.png?k=phylobox-1-0-ecfa61d1-db2b-49a3-9c05-8f4f682e68d9" /><br /><br />is just,<br /><br /><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em; max-width:300px;overflow:auto;"><br /><img src="http://phylobox.appspot.com/api/image.png?k=phylobox-1-0-ecfa61d1-db2b-49a3-9c05-8f4f682e68d9" style="max-width:300px; max-height:300px;" /><br /></pre><br /><br />Or obviously you can just download the image and host it from your own site.<br /><br /><b>Step 5: Romance them with the framework, steal their hearts with the canvas</b><br />Displaying data on the canvas is easy, displaying information on the canvas is less easy, adding interaction and user control to that information is downright nutty fun. I used my previous experience drawing phylogenies (<a href="http://geophylo.appspot.com/">GeoPhylo</a>) to do a first pass implementation to just display a tree on the canvas. Following that, Sander and I had a couple of build up sessions to figure out what exactly Version 1.0 should include. It was a lot. Primarily though, we wanted several different view types: Phylogram, Dendrogram, Circular-Dendrogram, and 3D. We also wanted to be able to display very big trees in useful ways that transition to more the more familar views (dendrogram or phylogram) when only a small part is within view. And we wanted a handful of tools to be ready from the get-go, but the ability to add a lot more as we move forward.<br /><br />Sander began writing a rendering engine for the trees. The idea was one engine for all view types, 3D or 2D. A few days later I went by his house for a beer and he showed me the engine at work. At that point he was trying to push 1000 node+branch objects into the canvas, spinning, at up to 60fps with shading. It was on.<br />We toned it back for display on our site to ~12fps and in a widget to just 8fps. This will keep a blog post with a couple tree widgets from completely sucking up all your CPU. Then engine is incredibly efficient though. We can render the tree in 3D in your browser and give you complete control over the editing tools and display of that tree. If you would like to see an example of a big tree, see the Frost et al. (2006) Amphibian tree (see phyloxml.org) rendered in your browser, <a href="http://phylobox.appspot.com/tree/edit?k=phylobox-1-0-b6c5201d-b7d0-4e60-8ee5-480811cd2221#wrapper">here</a><br /><br />Another exciting thing we did was add support for URIs in the nodes. This is built into phyloXML and we incorporated it into our phyloJSON. By enabling the URI tool in the editor, a user can link from a node to a GenBank record, a video, an image or anything you like. <a href="http://phylobox.appspot.com/tree/edit?k=phylobox-1-0-ecfa61d1-db2b-49a3-9c05-8f4f682e68d9#wrapper">This tree</a> has a few URI nuggets for you to discover (you must enable the URI tooltip and select the primary URI first).<br /><br />The engine also allows for multiple tree objects on the same canvas, although we haven’t unlocked this feature on the site yet.<br /><br /><b>Conclusion: PhyloBox</b><br /><br />We made PhyloBox flexible enough that you can use it from afar to just display your data or you can dive in and use all these features. We have already been thinking of some other important features to add if people like it. For example, why draw your tree once for publication and once for the web? We are working on export of high quality TIFFs or SVGs directly from the browser, for now PNGs will have to do. You like that? Well how about publishing a link to your interactive tree right in your paper? How about publishing the Widget itself right in a PlosOne paper? You don’t want people to fork your project? We are thinking about adding an opt-out option, but frankly don’t like it, it seems anti-scientific. We would like to hear your feedback on this one, maybe we could let you opt-out for only a limited amount of time (say 1 year) for a tree? You want to develop something home-grown to display a tree you customized on PhyloBox? You got it, use our <a href="http://phylobox.appspot.com/api/lookup">lookup service</a> to just retrieve the PhyloJSON object for one of your trees. Want commenting on trees? We can do that, just need to know the demand. We were thinking of having commenting on individual nodes, branches, or trees. I know having only phyloXML support can be a drag, I should have at least a Newick converter and upload available in the next week.<br />We really hope people like the PhyloBox concept. If not, we’d like to hear that too, because we need to know if you want us to develop this into the future. PhyloBox is somewhere around Beta, but the iEvoBio deadline is right now, so that is that. There is so much more we can add to this!<br /><br /></div>Andrew W Hillhttp://www.blogger.com/profile/04814436364156670727noreply@blogger.com7tag:blogger.com,1999:blog-5286956948868856311.post-60317560333073346902010-04-14T12:43:00.000+02:002010-04-14T12:43:53.794+02:00FOSS4G and Biodiversity International Year<div class="separator" style="clear: both; text-align: center;"><a href="http://media.tumblr.com/tumblr_l0477dxdiD1qabv21.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="93" src="http://media.tumblr.com/tumblr_l0477dxdiD1qabv21.png" width="320" /></a></div><br />
I am not sure if there is enough people on the biodiversity informatics/conservation community that knows about the great conference on Open Source and GIS <a href="http://2010.foss4g.org/">FOSS4G</a>.<br />
<br />
Most of us are already using great Open Source software for GIS. For example Geoserver, PostGIS, Quantum GIS,gvSIG, GRASS, OpenLayers, GDAL/OGR, MapServer and many more. This is the conference were all those developers meet. The conference include Workshops, tutorials and presentations. The level of the program is just amazing. Let me show you some of the ones of my own interest:<br />
<br />
<span class="Apple-style-span" style="color: #777777; font-family: arial, sans-serif; font-size: 14px; line-height: 18px;"><b style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><a href="http://www.blogger.com/tutorial08.php" style="color: #00669e; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-decoration: underline;">T-08: How to generate billions of tiles using distributed cloud-computing</a></b></span><br />
<span class="Apple-style-span" style="color: #777777; font-family: arial, sans-serif; font-size: 14px; line-height: 18px;"><b style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><a href="http://www.blogger.com/tutorial11.php" style="color: #00669e; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-decoration: underline;">T-11: Standardized geoprocessing with 52°north open source software</a></b></span><br />
<span class="Apple-style-span" style="color: #777777; font-family: arial, sans-serif; font-size: 14px; line-height: 18px;"><b style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><a href="http://www.blogger.com/workshop14.php" style="color: #00669e; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-decoration: underline;">W-14: Practical introduction to GRASS</a></b></span><br />
<span class="Apple-style-span" style="color: #777777; font-family: arial, sans-serif; font-size: 14px; line-height: 18px;"><b style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><a href="http://www.blogger.com/workshop09.php" style="color: #00669e; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px; text-decoration: underline;">W-09: Quantum GIS and PostGIS: Solving spatial problems and creating web-based analysis tools</a></b></span><br />
<br />
Or take a look at <a href="http://2009.foss4g.org/presentations/">last year presentations</a> to get excited on the great topics that are discussed on this conference.<br />
<br />
So I really would like you to consider attending this great conference and participate on the Open Source movement on GIS.<br />
<br />
But more important. In this International year on Biodiversity, I would like to get something organized around Biodiversity/Conservation and Open Source Geospatial software. I truly believe that GIS Open Source software is enabling the development of lot of our initiatives, and without it handling biodiversity knowledge would just be impossible.<br />
<br />
<br />
The idea is to have enough people attending from this community to create a side event specific to biodiversity. Last year for example they created one event around interoperability of projects on Climate<br />
Change. What about something similar for biodiversity?<br />
<br />
The deadline for paper submissions and presentations is in 2 days, I know it is tight, and it would help me a lot to get support from the organizers if we get a decent submission on biodiversity/conservation<br />
topics.<br />
<br />
Considering that Biodiversity is deeply linked to Geospatial information I think it could be a great venue to push the requirements from our community into the development of Open Source GIS that most<br />
of us are using anyway.<br />
<br />
I have contacted organizers and they are willing to organize a side event for Biodiversity/Conservation, but they just need to know there is enough people interested on it.<br />
<br />
I have sent multiple messages to different people and list that I know could be interested and I am getting already lot of answers of people that are gonna be submitting abstracts. I will try to collect all of them in this post so that we can see how the community will be represented.<br />
<br />
Additionally if it is possible for you consider supporting the event as a sponsor or providing an official letter of support for the initiative.<br />
<br />
Finally. We still need to decide what kind of event we will like to see there. Presentations, an interoperability test (like last year <a href="http://2009.foss4g.org/ccip/">Climate Challenge Integration Plugfes</a>t)... we decide.<br />
<br />
Looking forward to see lot of people from the community in Barcelona!Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com9tag:blogger.com,1999:blog-5286956948868856311.post-45588076695914568292009-12-15T22:10:00.001+01:002009-12-15T22:48:31.075+01:00Recreating the European Starling story<div class="separator" style="clear: both; text-align: center;"><a href="http://upload.wikimedia.org/wikipedia/commons/thumb/1/1b/European_Starling_2006.jpg/180px-European_Starling_2006.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="http://upload.wikimedia.org/wikipedia/commons/thumb/1/1b/European_Starling_2006.jpg/180px-European_Starling_2006.jpg" width="133" /></a><br />
</div>The European Starling is a bird native to most of temperate Europe and western Asia. I do not remember who was the first person that talked to me about it, Tim?, but since then I knew I wanted to develop something with it. The reason is funny/sad story behind it. The bird, that was only native in Europe and Asia, got introduced in lot of different places at the end of the 19th century. Since then it has spread to all continents and has been treated like a pest in lot of places, and for example, in Australia there is still efforts to prevent its introduction in the West. On the other hand in Europe it has been declining a lot, and it is now actually covered there by the Red List.<br />
<br />
Specially appealing is the story on how it got introduced in North America. I will just quote Wikipedia for that:<br />
<br />
<span style="font-family: sans-serif; font-size: 13px; line-height: 19px;"><i>Although there are approximately 200 million starlings in </i><a href="http://en.wikipedia.org/wiki/North_America" style="-webkit-background-clip: initial; -webkit-background-origin: initial; background-attachment: initial; background-color: initial; background-image: none; background-position: initial initial; background-repeat: initial; color: #002bb8; text-decoration: none;" title="North America"><i>North America</i></a><i>, they are all descendants of approximately 60 birds (or 100 </i><a class="external autonumber" href="http://www.birds.cornell.edu/AllAboutBirds/BirdGuide/European_Starling.html" rel="nofollow" style="-webkit-background-clip: initial; -webkit-background-origin: initial; background-attachment: initial; background-color: initial; background-image: url(http://en.wikipedia.org/skins-1.5/monobook/external.png); background-position: 100% 50%; background-repeat: no-repeat; color: #3366bb; padding-bottom: 0px; padding-left: 0px; padding-right: 13px; padding-top: 0px; text-decoration: none;"><i>[1]</i></a><i>) released in 1890 in </i><a href="http://en.wikipedia.org/wiki/Central_Park" style="-webkit-background-clip: initial; -webkit-background-origin: initial; background-attachment: initial; background-color: initial; background-image: none; background-position: initial initial; background-repeat: initial; color: #002bb8; text-decoration: none;" title="Central Park"><i>Central Park</i></a><i>, </i><a href="http://en.wikipedia.org/wiki/New_York" style="-webkit-background-clip: initial; -webkit-background-origin: initial; background-attachment: initial; background-color: initial; background-image: none; background-position: initial initial; background-repeat: initial; color: #002bb8; text-decoration: none;" title="New York"><i>New York</i></a><i>, by </i><a href="http://en.wikipedia.org/wiki/Eugene_Schieffelin" style="-webkit-background-clip: initial; -webkit-background-origin: initial; background-attachment: initial; background-color: initial; background-image: none; background-position: initial initial; background-repeat: initial; color: #002bb8; text-decoration: none;" title="Eugene Schieffelin"><i>Eugene Schieffelin</i></a><i>, who was a member of the Acclimation Society of North America reputedly trying to introduce to North America every bird species mentioned in the works of </i><a href="http://en.wikipedia.org/wiki/William_Shakespeare" style="-webkit-background-clip: initial; -webkit-background-origin: initial; background-attachment: initial; background-color: initial; background-image: none; background-position: initial initial; background-repeat: initial; color: #002bb8; text-decoration: none;" title="William Shakespeare"><i>William Shakespeare</i></a><i>.</i></span><br />
<span style="font-family: sans-serif; font-size: small;"><span style="font-size: 13px; line-height: 19px;"><br />
</span></span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://vimeo.com/8202268"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVVTZsOPAEMqfvK1QiREEttGxuySCR77S0JviiraOIZRBuWQLQvnXRT39sdOjP9bGPyyWP10RDoA2AY1lOPK_W9GRVIITpMiaAHCVaivNyZpM6OzotQBigSEhuIW3sturRpI0NdC7ZostH/s200/Screen+shot+2009-12-15+at+9.11.48+PM.png" /></a><br />
</div><span style="font-family: sans-serif; font-size: small;"><span style="font-size: 13px; line-height: 19px;"><span style="font-family: Times; font-size: medium; line-height: normal;">I knew this was a story that could really be catchy. Specially if we could use scientific primary data to show this story. While working with Tim Robertson and Andrew Hill we started thinking about using <a href="http://code.flickr.com/blog/tag/clustr/">Clustr</a>, from Flickr, to create polygons out of primary data and see if we could display this story. I demoed this in Geoweb and TDWG this year and the feedback was most of the time really good. You can <a href="http://vimeo.com/8202268">watch the video at Vimeo</a>.</span></span></span><br />
<br />
The challenge for that was that there is more than 1 million observations of the starling now available on GBIF and the classical point in map did not work well, the visualizations were tedious... well, kind of complicate. But the second semester of this year we started to see <a href="http://www.mysociety.org/2007/more-travel-maps/">interactive maps that seemed to be analyzing raster images on the fly</a> in Flash. This is really really cool. And since then we were just thinking more and more in raster representation of data to further filter in the client and allowing much more rich story telling. And then, one day, I showed the work from Andrew Cottam from WCMC on sea level rise and Google Maps for Flash. That was awesome! And being such a nice guy he is, he publish his code and saved me the time of figuring out the bitwise operations needed for at least one band raster. I am not sure if he wants me to put a link to his ongoing work so I will wait for him to publish it first (maybe in this blog ;) ).<br />
<br />
So I could not resist and with the help of Tim preparing the raster tiles for the starling, and Sergio doing some UI, we prepared the following demo application.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://vizzuality.s3.amazonaws.com/starling/index.html"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjq-riYNuhkQkwpdGZoPxmo0fLzckbkizFOtr23lsQ8DhXEO-Lufq1qQx2SjRP7QLcChyphenhyphen6TxEm4fE15fu40IE65rkBt0e6bR3I3uLGyZ8tQ8wBt4U12LfQ5FlC03n8b4dTqojfqQ6OGluBI/s640/Screen+shot+2009-12-15+at+9.37.11+PM.png" /></a><br />
</div>(Click the image to open)<br />
<br />
Drag the slider from 1880 to 2010 to see the accumulative records (by date recorded) for the data available on the GBIF network. While you drag the slider you will be presented with tooltips mostly taken from Wikipedia.<br />
<br />
Soon we will release all the source code, once a bit cleaned, and will share more technical details. And the best is yet to come... we only used one band on this demo, but we have 3 to play with!!<br />
<br />
I hope you like it and want to share some comments.<br />
<br />
Ah! Dont forget to turn on sound!Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com6tag:blogger.com,1999:blog-5286956948868856311.post-81008656204260839592009-11-16T07:49:00.012+01:002009-11-17T07:09:54.728+01:00Amazon EC2, EBS RAID-0 & PostGIS build script<div><span class="Apple-style-span" style="font-size:large;">EC2's dirty secret</span></div><a href="http://biodivertido.blogspot.com/2009/10/install-postgresql-84-and-postgis-140.html">Javier's post</a> was a great tutorial on building out a PostGIS database on Amazon EC2. We all know EC2, but it does have it's drawbacks and they are mainly related to disk IO. When using EC2 & EBS with large datasets you can easily run into IO bottlenecks. Individually these are not such a big deal, but when you are conducting global analyses poor disk IO on EC2 & EBS can quickly become a problem.<div><br /></div><div><span class="Apple-style-span" style="font-size:large;">Clean living?</span></div><div>To help alleviate this, there is a <a href="http://www.mysqlperformanceblog.com/2009/08/06/ec2ebs-single-and-raid-volumes-io-bencmark/">trend</a> <a href="http://tech.backtype.com/increasing-io-performance-on-ec2ebs">of</a> <a href="http://alestic.com/2009/06/ec2-ebs-raid">people</a> <a href="http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/">stringing</a> together EBS volumes and creating their own software RAID-0 arrays to achieve higher read and write throughput.</div><div><br /></div><div><span class="Apple-style-span" style="font-size:large;">Nope, a Bash script.</span></div><div>I pieced together bits and bobs to create a script that builds out a PostGIS database on an n-volume RAID array on EC2. It's pretty simple stuff, but should mean that instead of hours, you can get your 20 volume RAID-0 PostGIS test rig up and running in minutes. </div><div><br /></div><div>You can grab it from Github:</div><div><br /></div><div><a href="http://github.com/tokumine/ebs_raid_postgis">http://github.com/tokumine/ebs_raid_postgis</a></div>Anonymousnoreply@blogger.com5tag:blogger.com,1999:blog-5286956948868856311.post-47427714084716419762009-11-09T02:31:00.004+01:002009-11-09T03:15:38.428+01:00Automated informatics pipelines, public datasets, and the cloudAlthough Biodivertido typically focuses on <span style="font-style: italic;">biodiversity</span> informatics, I'm going to step it back to informatics of another stripe in the hopes of making a few points and getting people primed for Javier's TDWG talk <a href="http://www.tdwg.org/conference2009/program/">Moving Biodiversity to the Cloud</a> on Friday.<br /><br />A couple of days ago I became curious if I could use <a href="http://sourceforge.net/apps/mediawiki/cloudburst-bio/index.php?title=CloudBurst">CloudBurst</a> to compare novel influenza sequences to the entire known sequence record of influenza in the hopes of detecting novel <a href="http://en.wikipedia.org/wiki/Reassortment">reassortants</a> on the fly. The intended use of CloudBurst is to take small sequence reads from high throughput sequencing projects and query them against a complete genome a variety of reasons including <span style="font-style: italic;">SNP discovery, genotyping and personal genomics</span><a href="http://bioinformatics.oxfordjournals.org/cgi/content/abstract/btp236"><span style="vertical-align: top;font-size:85%;" >1</span></a>. So why not take a novel influenza and break it up into many small overlapping fragments and query them against a genome comprised of all known sequences to detect reassortment? <p>Comparisons similar to these can likely be done using some manipulation of the <a href="http://www.ncbi.nlm.nih.gov/Education/BLASTinfo/BLAST_algorithm.html">Blast</a> or similar algorithm. But in my case, I was particularly interested in CloudBurst because of its available implementation in EC2 using Hadoop to utilise the cluster environment. This means that if successful I could later wrap my method into an automated workflow for detecting and reporting potentially novel influenza reassortants as soon as new sequences are reported.</p> <p>I posted an overview of the steps it took below. In the end, as a first pass detection of reassortment it appears promising or at least interesting. All in all, I went from idea, to data assembly, to trial implementation in about 2 hours (not counting a few hours trying to get Hadoop to run the CloudBurst.jar. Hadoop version number = VERY important). That is great for the study of influenza, but what about biodiversity informatics?</p> <p>There are a few things that made this so simple. </p> <p>First, a rapid adoption of EC2 by the bioinformatics community. Look for example at the JVCI maintained <a href="http://www.jcvi.org/cms/research/projects/jcvi-cloud-biolinux/">Bio-linux distribution</a> for use on EC2. It comes prepackaged with a whole smörgåsbord of very fun little tools. It costs 34 cents per hour (not 10 because it requires a 64-bit instance) and takes about 1 minute to set up your very own copy. Where is our (the <span style="font-style: italic;">biodiversity</span> informatics community's) GISing, niche modeling, distribution mapping, PDing, SGAing, Hadoop spatial joining, machine image? </p> <p>Second, a growing community of users who are exchanging knowledge about how to tackle these very large datasets in the cloud. Biodiversity informatics is growing its own such community, writers of this blog can attest to that.</p> <p>Third, cloud hosted datasets. Amazon Web Services is making it fairly simple to host very large datasets that anyone in the community can access. The <a href="http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2261">GenBank</a> image is ~200gb and takes 5 minutes to set up as a mounted volume for my own use. This targets a completely different consumer then our data portals and APIs. It opens up the data for the community of informatics enthusiasts doing cool things on the cloud (think spelling correction, retrospective georeferencing, spatial joining). </p> <p>With all that said I'd like to return to think about how this relates to biodiversity informatics. I've been talking with a variety of people over the past few weeks and all of them show some level of interest in moving to the cloud. What I'm worried about now is that we will adopt many of the same stances as a community as we have had without the cloud. The case study I developed above relied heavily on easily accessible data and tools. We as a community must move forward with this as a primary goal. Likely, there will always be a need for portals and APIs, but for really big questions sometimes it is just easier to have the entire dataset ready for access. Why hasn't our community got it together enough to launch a unified public dataset in the cloud? </p> <p>I guess data quality concerns and ownership are two primary concerns. I'm sorry, but those are bad reasons, time to grow up, it isn't 2001 any longer.<br /></p> <p>Once in place we can begin to build new methods (or reapply existing ones) for parsing out duplicate records, linking data to geographic areas, merging error types into targeted datasets, and sharing findings with the owners of the original data. The 'snapshot' approach as implemented in AWS Public Data Set makes it so we consumers constantly rely on the original providers to include the newest and most up to date records, none of our hard work will be included in future snapshots if we don't come up with methods for reporting corrections to the source. </p> <p>It is important to restate, I don't think portals and APIs will go away. They are for two completely different consumer communities than those interested in looking at the entire dataset at once, versus pulling smaller subsets of data manually or developing tools to do so via API interfaces. I do think that by providing public datasets, new methods and technologies for enhancing the portals and APIs will arise, as well as still unknown methods for improving the datasets at source, and ultimately enhancing our knowledge about the world's biodiversity.<br /></p> <p></p><p><br /></p><span style="font-weight: bold;">METHODS</span> <p>After downloading Hadoop (version 0.18.3), here is what I ran.</p><p>Launch Hadoop cluster, create and mount a volume containing the <a href="http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2419&categoryID=246">GenBank influenza data</a><br /></p> <blockquote><p><span style="font-size:85%;">>src/contrib/ec2/bin/hadoop-ec2 launch-cluster cloud 5</span></p><p><span style="font-size:85%;">>ec2-create-volume --snapshot snap-fe3ec297 -z us-east-1d</span></p> <p><span style="font-size:85%;">>ec2-attach-volume vol-7b49a612 -i <master-node-id>MASTERNODEIDHERE -d /dev/sdh</master-node-id></span></p> <p><span style="font-size:85%;">>src/contrib/ec2/bin/hadoop-ec2 login cloud</span></p> <p><span style="font-size:85%;">$mkdir /inf </span></p> <p><span style="font-size:85%;">$mount /dev/sdh /inf</span></p></blockquote><p></p> <p>Here I ran a small Python script to extract all sequences (hemagluttanin only) known prior to the original swine flu outbreak, concatenate those sequences into a single “genome” saved as data/genome.fa, and recording a map of where in the genome each sequence ended.</p> <p>Next, a group of sequences (again, hemagluttanin only) from the earliest swine flu outbreaks were broken up into numerous overlapping fragments and saved as data/segs.fa, again keeping a map of where each fragment belonged. These are the 'novel sequences' I would test for any cases of reassortment.</p> <p>Download and extract CloudBurst</p> <p></p><blockquote><p><span style="font-size:85%;">$wget http://downloads.sourceforge.net/project/cloudburst-bio/cloudburst/CloudBurst-1.0.1/CloudBurst-1.0.1.tgz</span></p> <p><span style="font-size:85%;">$tar xzf CloudBurst-1.0.1.tgz</span></p> <p><span style="font-size:85%;">$mv CloudBurst-1.0.1/ data/</span></p></blockquote><p></p> <p>Convert fasta files to Hadoop ready files</p> <p><span style="font-size:85%;"></span></p><blockquote><p><span style="font-size:85%;">$java -jar data/ConvertFastaForCloud.jar data/genome.fa data/genome.br</span></p> <p><span style="font-size:85%;">$java -jar data/ConvertFastaForCloud.jar data/segs.fa data/segs.br</span></p></blockquote><p><span style="font-size:85%;"></span></p> <p>Move the data to hdfs</p> <p></p><blockquote><span style="font-size:85%;">$/usr/local/hadoop-0.17.0/bin/hadoop fs -put ~/data /data</span></blockquote><p></p> <p>Run the analysis</p> <p><span style="font-size:85%;"><blockquote>$/usr/local/hadoop-0.17.0/bin/hadoop jar ~/data/CloudBurst.jar /data/genomeA.br /data/segsA.br /results 40 3 0 1 50 15 5 5 128 5 </blockquote></span></p> <p>Copy results from hdfs</p> <p></p><blockquote><span style="font-size:85%;">$/usr/local/hadoop-0.17.0/bin/hadoop fs -get /results results</span></blockquote><p></p> <p>Convert the results back to human readable format</p> <p><span style="font-size:85%;"><blockquote>java -jar data/PrintAlignments.jar results >results.txt</blockquote></span></p> <p>Profit!</p> <p>Parsing meaning out of the results was a bit more labor intensive and I forewent any automation (for now), using OpenOffice spreadsheets to map matched portions back to their original viruses and report accession numbers. What I found was interesting, exciting and promissing, even matching some of the findings reported in <a href="http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0006402">Kingsford et al., 2009</a>, from only 2 hours of work and a couple of dollars. I was pretty satisfied.</p>Andrew W Hillhttp://www.blogger.com/profile/04814436364156670727noreply@blogger.com0tag:blogger.com,1999:blog-5286956948868856311.post-41899369853572906212009-10-21T19:54:00.001+02:002009-11-09T17:22:55.997+01:00Install PostgreSQL 8.4 and PostGIS 1.4.0 in Ubuntu 9.0.4I am a big fan of the new PostGIS 1.4.0 (and also of <a href="http://www.youtube.com/watch?v=zB_a28vBtBk&feature=channel">Paul Ramsey</a>) . I always have troubles installing PostGIS in ubuntu so I thought that this time I was gonna document it and blog it here. So this is just a log of the steps required to install it on an EC2 instance with Ubuntu 9.04. I hope it can be useful for someone else.<br />
<br />
Just for the record. The EC2 instance I used was <span style="font-family: arial, sans-serif; font-size: 13px;"><b>ami-ccf615</b> from <a href="http://alestic.com/" rel="nofollow">http://alestic.com</a> .</span><br />
<span style="font-family: arial, sans-serif; font-size: small;"><span style="font-size: 13px;"><br />
</span></span><br />
<span style="font-family: arial, sans-serif; font-size: small;"><span style="font-size: 13px;">Once login (totally fresh).</span></span><br />
<span style="font-family: arial, sans-serif; font-size: small;"><span style="font-size: 13px;"><br />
</span></span><br />
<span style="font-family: arial, sans-serif; font-size: small;"><span style="font-size: 13px;"></span></span><br />
<span style="font-family: arial, sans-serif; font-size: small;"></span><br />
<span style="font-family: arial, sans-serif; font-size: small;"><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em;"><span style="color: black;">apt</span><span style="color: #666600;">-</span><span style="color: #000088;">get</span><span style="color: black;"> update
apt</span><span style="color: #666600;">-</span><span style="color: #000088;">get</span><span style="color: black;"> install vim</span></pre><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em;"><span style="color: black;">#The sources are still not available on the regular package servers... edit the sources
vim </span><span style="color: #666600;">/</span><span style="color: black;">etc</span><span style="color: #666600;">/</span><span style="color: black;">apt</span><span style="color: #666600;">/</span><span style="color: black;">sources</span><span style="color: #666600;">.</span><span style="color: black;">list
add deb http</span><span style="color: #666600;">:</span><span style="color: #880000;">//ppa.launchpad.net/pitti/postgresql/ubuntu jaunty main</span><span style="color: black;">
deb</span><span style="color: #666600;">-</span><span style="color: black;">src http</span><span style="color: #666600;">:</span><span style="color: #880000;">//ppa.launchpad.net/pitti/postgresql/ubuntu jaunty main</span><span style="color: black;">
sudo apt</span><span style="color: #666600;">-</span><span style="color: black;">key adv </span><span style="color: #666600;">--</span><span style="color: black;">keyserver keyserver</span><span style="color: #666600;">.</span><span style="color: black;">ubuntu</span><span style="color: #666600;">.</span><span style="color: black;">com </span><span style="color: #666600;">--</span><span style="color: black;">recv</span><span style="color: #666600;">-</span><span style="color: black;">keys </span><span style="color: #006666;">8683D8A2</span><span style="color: black;">
sudo apt</span><span style="color: #666600;">-</span><span style="color: #000088;">get</span><span style="color: black;"> update
sudo apt</span><span style="color: #666600;">-</span><span style="color: #000088;">get</span><span style="color: black;"> install postgresql</span><span style="color: #666600;">-</span><span style="color: #006666;">8.4</span></pre><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em;"><span style="color: black;"><span style="color: #006666;"><span style="color: blue;">#This changes the port from 5433 to 5432</span>
</span>sudo sed </span><span style="color: #666600;">-</span><span style="color: black;">i</span><span style="color: #666600;">.</span><span style="color: black;">bak </span><span style="color: #666600;">-</span><span style="color: black;">e </span><span style="color: #008800;">'s/port = 5433/port = 5432/'</span><span style="color: black;"> </span><span style="color: #666600;">/</span><span style="color: black;">etc</span><span style="color: #666600;">/</span><span style="color: black;">postgresql</span><span style="color: #666600;">/</span><span style="color: #006666;">8.4</span><span style="color: #666600;">/</span><span style="color: black;">main</span><span style="color: #666600;">/</span><span style="color: black;">postgresql</span><span style="color: #666600;">.</span><span style="color: black;">conf
sudo </span><span style="color: #666600;">/</span><span style="color: black;">etc</span><span style="color: #666600;">/</span><span style="color: black;">init</span><span style="color: #666600;">.</span><span style="color: black;">d</span><span style="color: #666600;">/</span><span style="color: black;">postgresql</span><span style="color: #666600;">-</span><span style="color: #006666;">8.4</span><span style="color: black;"> stop
sudo </span><span style="color: #666600;">/</span><span style="color: black;">etc</span><span style="color: #666600;">/</span><span style="color: black;">init</span><span style="color: #666600;">.</span><span style="color: black;">d</span><span style="color: #666600;">/</span><span style="color: black;">postgresql</span><span style="color: #666600;">-</span><span style="color: #006666;">8.4</span><span style="color: black;"> start
apt</span><span style="color: #666600;">-</span><span style="color: #000088;">get</span><span style="color: black;"> install postgresql</span><span style="color: #666600;">-</span><span style="color: black;">server</span><span style="color: #666600;">-</span><span style="color: black;">dev</span><span style="color: #666600;">-</span><span style="color: #006666;">8.4</span><span style="color: black;"> libpq</span><span style="color: #666600;">-</span><span style="color: black;">dev
apt</span><span style="color: #666600;">-</span><span style="color: #000088;">get</span><span style="color: black;"> install libgeos</span><span style="color: #666600;">-</span><span style="color: black;">dev
wget http</span><span style="color: #666600;">:</span><span style="color: #880000;">//postgis.refractions.net/download/postgis-1.4.0.tar.gz</span><span style="color: black;">
apt</span><span style="color: #666600;">-</span><span style="color: #000088;">get</span><span style="color: black;"> install proj
tar xvfz postgis</span><span style="color: #666600;">-</span><span style="color: #006666;">1.4</span><span style="color: #666600;">.</span><span style="color: #006666;">0.tar</span><span style="color: #666600;">.</span><span style="color: black;">gz
cd postgis</span><span style="color: #666600;">-</span><span style="color: #006666;">1.4</span><span style="color: #666600;">.</span><span style="color: #006666;">0</span><span style="color: black;">
</span><span style="color: #666600;">./</span><span style="color: black;">configure
make
make install
sudo su postgres</span></pre><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em;"><span style="color: black;"><span style="color: blue;">#change the postgres password to "atlas" so that you can later login</span>
</span><span style="color: black;">psql -c"ALTER user postgres WITH PASSWORD 'atlas'"
createdb geodb </span><span style="color: #666600;">(</span><span style="color: #000088;">with</span><span style="color: black;"> password atlas</span><span style="color: #666600;">)</span><span style="color: black;">
createlang </span><span style="color: #666600;">-</span><span style="color: black;">dgeodb plpgsql
psql </span><span style="color: #666600;">-</span><span style="color: black;">dgeodb </span><span style="color: #666600;">-</span><span style="color: black;">f </span><span style="color: #666600;">/</span><span style="color: black;">usr</span><span style="color: #666600;">/</span><span style="color: black;">share</span><span style="color: #666600;">/</span><span style="color: black;">postgresql</span><span style="color: #666600;">/</span><span style="color: #006666;">8.4</span><span style="color: #666600;">/</span><span style="color: black;">contrib</span><span style="color: #666600;">/</span><span style="color: black;">postgis</span><span style="color: #666600;">.</span><span style="color: black;">sql
psql </span><span style="color: #666600;">-</span><span style="color: black;">dgeodb </span><span style="color: #666600;">-</span><span style="color: black;">f </span><span style="color: #666600;">/</span><span style="color: black;">usr</span><span style="color: #666600;">/</span><span style="color: black;">share</span><span style="color: #666600;">/</span><span style="color: black;">postgresql</span><span style="color: #666600;">/</span><span style="color: #006666;">8.4</span><span style="color: #666600;">/</span><span style="color: black;">contrib</span><span style="color: #666600;">/</span><span style="color: black;">spatial_ref_sys</span><span style="color: #666600;">.</span><span style="color: black;">sql
psql -dgeodb -c"select postgis_lib_version();"</span></pre><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em;"><span style="color: blue;">#This should return 1.4.0</span></pre><pre class="prettyprint" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 3px; font-size: 12px; margin-left: 2em; padding-bottom: 0.5em; padding-left: 0.5em; padding-right: 0.5em; padding-top: 0.5em;"><span style="color: #000088;">exit</span></pre><span style="font-family: Times;"></span><br />
<span style="font-family: Times;"><div><br />
</div></span></span><span style="font-family: arial, sans-serif; font-size: small;"><span style="font-family: Times;">Good Luck!</span></span>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com13tag:blogger.com,1999:blog-5286956948868856311.post-73030672177009817892009-09-20T21:06:00.004+02:002009-09-20T21:39:18.446+02:00Using Geowebcache tiles stored in s3 from Google Maps for FlashLast week I was in a code sprint for my next project on a very <a href="http://twitpic.com/i893h">nice cottage</a>. During the sprint I set up a <a href="http://geoserver.org">Geoserver</a> instance with <a href="http://geowebcache.org/trac">Geowebcache</a> to generate tiles from the World Database on Protected Areas. The tiles will be stored in S3 and distributed with CloudFront as the main goal is speed worldwide. The tiles will be used in a set of Maps in Flash around the project.<div><br /></div><div>The biggest issue came when trying to generate a class to load the tiles from the default folder structure created by Geowebcache. I know I could have created my own tile folder structure generator for Geowebcache by creating my own <a href="http://geowebcache.org/trac/browser/trunk/geowebcache/src/main/java/org/geowebcache/storage/blobstore/file/FileBlobStore.java">BlobStore</a> and modifying the <a href="http://geowebcache.org/trac/browser/trunk/geowebcache/src/main/java/org/geowebcache/storage/blobstore/file/FilePathGenerator.java">FilePathGenerator</a> to fir my needs, buuut i am not a Java lover and thought that it might be some good reasons why the good people at Geowebcache had use the current one.</div><div>The problem is that the structure is not very obvious, and I can imagine that it is because you dont want to have too many files within one single folder or you will start hitting the file system limits.</div><div><br /></div><div>In order to use Google Maps for Flash you need to convert a request based on Zoom, X and Y tile coordinates, into a URL from where to get the tiles</div><div><br /></div><div>If you use Geowebcache for accessing your tiles this is not a problem, use their gmap service and it will deliver your tile from wherever is in the file system. The issue is that I uploaded my whole geowebcache into S3 so I needed from the Flash client to know where to look for a x,y,z tile. With the help of <a href="http://twitter.com/craigmmills">Craig Mills</a> we ported the <a href="http://geowebcache.org/trac/browser/trunk/geowebcache/src/main/java/org/geowebcache/storage/blobstore/file/FilePathGenerator.java">code on Geowebcache</a> to AS3.</div><div><br /></div><div>The trickiest part was figuring out that Geowebcache starts its tiles from bottom left, while Google starts from top left. Dont know why they made this decission, but it is a bit of a pain. So tile z:1,x:0,y:0 in google is z:1,x:0,y:1 in geowebcache.</div><div><br /></div><div>I thought it might be useful for someone that has similar needs so I have uploaded my classes <a href="http://dl.getdropbox.com/u/580074/geowebcachetile.zip">here</a> in case someone needs them.</div><div><br /></div><div>Just a bit more information. I use the following command to trigger the tiling process:</div><div><br /></div><div><span class="Apple-style-span" style="font-family: helvetica, arial, freesans, clean, sans-serif; font-size: 13px; line-height: 20px; ">curl -u geowebcache:secured -d “threadCount=01&type=reseed&gridSetId=<span class="caps" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; line-height: 1.4em; ">EPSG</span>:900913&format=image/png&zoomStart=00&zoomStop=08&minX=&minY=&maxX=&maxY=” http://localhost:8080/geowebcache/rest/seed/ppe:ppeblue</span></div><div><br /></div><div>Remove the empty tiles generated (by looking if they are smaller than 663bytes)</div><div><br /></div><div><span class="Apple-style-span" style="font-family: helvetica, arial, freesans, clean, sans-serif; font-size: 13px; line-height: 20px; ">find . -size -663c -type f -exec rm -f ‘{}’ \;</span></div><div><br /></div><div>Remove the empty folders</div><div><br /></div><div><span class="Apple-style-span" style="font-family: helvetica, arial, freesans, clean, sans-serif; font-size: 13px; line-height: 20px; ">find . -empty -type d -exec rm -rf ‘{}’ \;</span></div><div><span class="Apple-style-span" style="font-family:helvetica, arial, freesans, clean, sans-serif;font-size:100%;"><span class="Apple-style-span" style="font-size: 13px; line-height: 20px;"><br /></span></span></div><div>And finally upload (well, sync) to S3</div><div><br /></div><div><span class="Apple-style-span" style="font-family: helvetica, arial, freesans, clean, sans-serif; font-size: 13px; line-height: 20px; ">s3cmd sync —guess-mime-type —delete-removed —force —no-progress —acl-public /var/tiles/gwc/ppe_ppeblue/ s3://mybuket/ppe_ppeblue/</span></div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com1tag:blogger.com,1999:blog-5286956948868856311.post-65880803702330938822009-08-14T12:35:00.003+02:002009-08-14T13:25:19.580+02:00Presentation at Geoweb 09: Biodiversity: Where is important<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgan_aal5nzIernIENSCFDsNVgPS_4mLOsWc6rG4ihXBaKoSDFe1FK_cHH8x9kjidTur-a2K75w2bi0IBvjsrsgWNibeoW9DV9uEv6pV8iYzK7p_D3sbkXMJwpp3nzZzWyeCS7vIbO43pV9/s1600-h/logoGeoweb.gif"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 200px; height: 125px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgan_aal5nzIernIENSCFDsNVgPS_4mLOsWc6rG4ihXBaKoSDFe1FK_cHH8x9kjidTur-a2K75w2bi0IBvjsrsgWNibeoW9DV9uEv6pV8iYzK7p_D3sbkXMJwpp3nzZzWyeCS7vIbO43pV9/s200/logoGeoweb.gif" border="0" alt="" id="BLOGGER_PHOTO_ID_5369767256499766882" /></a><div>I was a invited speaker at this year <a href="http://geowebconference.org/">Geoweb 09</a> conference in Vancouver. This year the conference was focus on Cityscapes. Kind of trying to figure out how the geoweb could help on urban infrastructure on cities and things like that. It was curious that someone like me, focus mainly on biodiversity, got invited as a speaker there. But considering that cities have a huge effect on biodiversity, I think it was great to be there to offer some perspective on what we are doing in Biodiversity Informatics in relation with the geoweb. Hopefully some people got the idea of openness of this community, what primary data is and how are we sharing it. Here is my presentation on slideshare and links to videos on youtube (in 5 parts).</div><div><div style="width:425px;text-align:left" id="__ss_1795930"><br /><object style="margin:0px" width="425" height="355"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=geoweb09-090731110448-phpapp01&stripped_title=geoweb09"><param name="allowFullScreen" value="true"><param name="allowScriptAccess" value="always"><embed src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=geoweb09-090731110448-phpapp01&stripped_title=geoweb09" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><div style="font-size:11px;font-family:tahoma,arial;height:26px;padding-top:2px;"><span class="Apple-style-span" style="font-family:Georgia, fantasy;font-size:130%;"><span class="Apple-style-span" style="font-size: 16px;"><span class="Apple-style-span" style="font-family:tahoma, arial;font-size:100%;"><span class="Apple-style-span" style="font-size: 11px;"><br /></span></span></span></span></div></div></div><div><br /><object width="425" height="344"><param name="movie" value="http://www.youtube.com/v/UZAQNLfg5aQ&hl=es&fs=1&"><param name="allowFullScreen" value="true"><param name="allowscriptaccess" value="always"><embed src="http://www.youtube.com/v/UZAQNLfg5aQ&hl=es&fs=1&" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"></embed></object><br /><br /><br /></div><div>The conference was great and <a href="http://www.cbc.ca/canada/british-columbia/story/2009/07/29/bc-heat-wave-forecast.html">some people claim it was very hot</a>, but really, these people dont know what a hot summer is if they havent been in August in Madrid.</div><div>I specially enjoyed meeting with Aaron Cope from Flickr, Paul Ramsey from PostGIS team and lot of different people there.</div><div><br /></div><div>I might start getting into the discussion on how biodiversity informatics can better share their data to better play on the geoweb, specially since there is some great <a href="http://randommarkers.blogspot.com/2009/07/thoughts-on-geoweb-standards.html">discussion going</a> <a href="http://highearthorbit.com/geoweb-standards-your-thoughts/">on</a>. Related to why OGC does not work with us is the question on how we can more effectively share our data so that it better plays on the Geoweb.</div><div><br /></div><div>Finally I think it is interesting if more people from biodiversity informatics start getting into this issue and this conference is a great place to discuss them. So, hopefully next year I will not be the only one talking about biodiversity at Geoweb 10!</div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com0tag:blogger.com,1999:blog-5286956948868856311.post-64713300908772202442009-07-03T10:59:00.007+02:002009-07-25T02:44:41.965+02:00Biodiversity databases and OGC standards don't play well together<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://farm1.static.flickr.com/125/324174667_0c1640e178.jpg?v=0"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 200px; height: 127px;" src="http://farm1.static.flickr.com/125/324174667_0c1640e178.jpg?v=0" border="0" alt="" /></a><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://www.opengeospatial.org/themes/ogc/logo.png"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 144px; height: 73px;" src="http://www.opengeospatial.org/themes/ogc/logo.png" border="0" alt="" /><span class="Apple-style-span" style="color: rgb(0, 0, 0); -webkit-text-decorations-in-effect: none; "></span></a><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://www.opengeospatial.org/themes/ogc/logo.png"><span class="Apple-style-span" style="color: rgb(0, 0, 0); -webkit-text-decorations-in-effect: none; ">Some days ago Tim and I had some discussions about how to provide OGC services for biodiversity databases, like for example the </span></a><a href="http://www.groms.de/">Global Registry of Migratory Species</a>. This time the reason for the discussion to start again was the discovery of the new <a href="http://www.inspire-geoportal.eu/index.cfm/pageid/341">INSPIRE Geoportal Viewer.</a> For those who don't know it, the <a href="http://inspire.jrc.ec.europa.eu/">INSPIRE directive</a> is pushing the creation of a common infrastructure fo sharing geospatial data within Europe. They plan to do that by using Open Standards like the ones from the <a href="http://www.opengeospatial.org/">Open Geospatial Consortium (OGC)</a>.<div><br /></div><div>The typical use case they always describe with these Spatial Data Infrastructures (SDIs) is having a registry of services (Catalog Service) where a user can find geospatial data. The data is available through Web services, like <a href="http://en.wikipedia.org/wiki/Web_Map_Service">Web Map Server (WMS)</a> or <a href="http://en.wikipedia.org/wiki/Web_Feature_Service">Web Feature Service (WFS)</a>. With a </div><div>web or desktop GIS client you discover, connect to the services, display different layers, do some analysis, print results, etc. All by using open standards and mixing data from lot of different places.</div><div style="text-align: center;"><br /></div><div>The INSPIRE registry and Geoportal Viewer is the typical example. On the viewer you can select from a list of available services (registered on the catalog service) or use your WMS web service URL. </div><div><div style="text-align: center;"><br /></div><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 120px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZoxl4pklisIMSj1LXU59RWVoylanmDqXC9LCTRyyHxDBvFHQ5rZ0meD68UHaQahuFUgvwfx-OW5KeBH0CJKmyUBX9Ri1MsbtrxanfXukwDN-Bx2IHcqOh9kHzfZhNZX3594ecBEeIOteN/s400/Picture+11.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5354167586014235810" /></div><div style="text-align: left;">When you select a service, you get a list of available layers on it. For example in the Spanish SDI you get this:</div><div style="text-align: left;"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 316px; height: 181px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIazHgUhQXEnQz6tkYuW5EQmzo3DRsmC0RDWpfhtvusUfVe5-E9kO6t4DbIVFGp0FnOUZZMdQ33i4sPnBMQN-KffL7sursCWVz9V0v2sOad17lmsBIouf8vfkBWOlUSCE-1NnGRIZ1epLt/s400/Picture+12.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5354168362070939634" /></div><div style="text-align: left;">The way this work internally is by doing a getCapabilities request to the WMS server. The returned XML document list the layers available with metadata on how to query it, owner, etc.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">But what happens when you have a database like the GBIF cache with 1.8 Million species? You can not create a layer per species, or the getCapabitilites document will be MB and MB impossible to parse by any client. In any case, who wants to provide a list of 1.8M species to select a layer?</div><div style="text-align: left;"><br /></div><div style="text-align: left;">Well, the way to make it work is to specify a filter on the WMS request specifying for example the species_id you are interested in. But those generic clients do not support specifying these kind of filters.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">To me that means that the current status of OGC clients, like the ones used by INSPIRE, <a href="http://www.earthobservations.org/">GEOSS</a>, <a href="http://www.idee.es/show.do?to=pideep_pidee.ES">National SDIs</a>, etc. are not able to handle biodiversity OGC services. Or say it in a different way, OGC services are not prepared to handle biodiversity databases with lot of species.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">What are the possibilities? </div><div style="text-align: left;"><br /></div><div style="text-align: left;">1) OGC supports on their capabilities documents things like "Hey! I am not a service with a set of layers, I am a datastore with potentially millions of layers. So if you want to grab anything from me, you are going to provide a filter in your request". This will imply that OGC do some work and more important, software clients support this work. I think this will not happen in a few years.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">2) Create a set of interesting layers in biodiversity. We could match our biodiversity databases against <a href="http://www.iucnredlist.org/">IUCN list of endangered species</a>, create richness maps, etc, but access to primary data per species will not be possible.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">If you think on the potential customers we probably should be thinking on a predefined list of layers that we could all create on our OGC services that might be interesting for lot of people. Richness, endangered species, kingdoms, whatever...</div><div style="text-align: left;"><br /></div><div style="text-align: left;">Other possibility is that we create portals where the user filter for a species and then gets a "customized,dynamic GetCapabilities document" that will include the filter on the URL. That will be possible. But with Catalog Services, like <a href="http://geossregistries.info/">GEOSS</a>, where there will thousands of services, is biodiversity going to be so special as for the user to go to one of our website before continuing in their wonderful world of web services workflow? I doubt it.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">Next week I am going to <a href="http://geowebconference.org/">Geoweb 09</a> as invited speaker to talk about Biodiversity and the challenges to share it on the Geoweb. I would love to hear what do you think about using OGC services within our community or any other issue related to geospatial data and analysis.</div><div style="text-align: left;">I would love to hear what do you think about using OGC services within our community.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><span class="Apple-style-span" style="font-size: small;">(<a href="http://www.flickr.com/photos/mikebaird/324174667/">bird picture </a>by <span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif; font-size: 11px; color: rgb(102, 102, 102); "><b property="foaf:name"><a href="http://www.flickr.com/photos/mikebaird/" title="Link to mikebaird's photostream" rel="dc:creator cc:attributionURL" style="color: rgb(0, 99, 220); text-decoration: underline; ">mikebaird</a><span class="Apple-style-span" style="color: rgb(0, 0, 0); font-family: Georgia, fantasy; font-size: 16px; font-weight: normal; "><span class="Apple-style-span" style="font-size: small;">)</span></span></b></span></span></div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com7tag:blogger.com,1999:blog-5286956948868856311.post-69364367543289586992009-06-25T18:07:00.005+02:002009-06-29T10:28:14.503+02:00RSS feeds used by publishersOne of my current tasks is working on tools to index publications in order to find scientific names. One of the first things to figure out is how to discover publications. Many publishers provide various RSS feeds for their latest issue(s), a feature that <a href="http://www.ubio.org/index.php">uBio RSS</a> is making use of, scanning about 980 journal feeds as of today.<br /><br />I am trying to put some recommendations together for publishers on how to encode their RSS feeds or to use other formats to make their digital publications discoverable. If you have any recommendations I'd be glad to know about them. Especially on how to best promote back catalogues of all available publications would be interesting, as RSS feeds natively only show the latest ones (there are paging extensions for Atom, but that has no widespread support). Sitemaps or OAI-PMH seem like a good candidate, although something easier than OAI would be preferred.<br /><br />Wondering which RSS format is most widely used by publishers currently and which extensions they use to encode their metadata, I wrote a little tool today that reads all current feeds known to ubio and checks their rss format, here are the results, not analyzing the namespaces and extension formats yet:<br /><br /><pre><br />rss_0.92 = 3<br />rss_1.0 = 336<br />rss_2.0 = 431<br />rss_0.91U = 6<br />atom_1.0 = 2<br /></pre><br /><br />So clearly the rdf based rss 1.0 (often together with Prism) and the simple rss 2.0 format is used mostly.<br />If there only would be a simple way to page. Maybe <a href="http://msdn.microsoft.com/en-us/xml/bb190613.aspx">Microsofts Simple Sharing Extensions</a> could help?Anonymoushttp://www.blogger.com/profile/02525336976753861766noreply@blogger.com3tag:blogger.com,1999:blog-5286956948868856311.post-4955103672688962842009-06-25T11:01:00.002+02:002009-06-25T11:03:48.509+02:00Visualizing Tweeter biodiversity observations<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDNbKdu5yz3HnDfleiQvL6Scm64x0f04hC2xNbieRluG38w_2pfRYlTqfKZZjM4HOzlaBr9jmQJiG6gYu8isyktfV7UYWrXPohMYdSBz-2nVuKqS6c6m4Z9jnjI8GLfsxJwhrzFajb-GbG/s1600-h/Picture+7.png"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 195px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDNbKdu5yz3HnDfleiQvL6Scm64x0f04hC2xNbieRluG38w_2pfRYlTqfKZZjM4HOzlaBr9jmQJiG6gYu8isyktfV7UYWrXPohMYdSBz-2nVuKqS6c6m4Z9jnjI8GLfsxJwhrzFajb-GbG/s320/Picture+7.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5351188235679499426" /></a>At <a href="http://www.e-biosphere09.org/">Ebiosphere09</a>, there was an <a href="http://www.e-biosphere09.org/page/30/the-e-biosphere-09-informatics-challenge.html">Informatics Challenge</a> that <a href="http://iphylo.blogspot.com/2009/05/e-biosphere-challenge-visualising.html">Rod Page won</a>, congratulations!<div>Vizzuality wanted to participate but we did not find the time to work on it. During the conference we found people like <a href="http://twitter.com/IvyMan">@IvyMan</a> twittering observations on biodiversity. This was part of the "<a href="http://spire.umbc.edu/ebio/">The eBiosphere Real-Time Citizen Science Challenge!</a>" which published the rules on how to tweet.</div><div><br /></div><div>We were far too late to participate on it, but we thought it could be cool to give it a try using the new Flex 4 "Gumbo". <a href="http://twitter.com/xavijam">@xavijam</a> from Vizzuality was starting to learn Flex 4 so he took this challenge.</div><div><br /></div><div>We make use of the Twitter API to query for the patterns explained on the challenge rules. Once we get the tweets, parse the latitude and longitude, get the scientific name of the observation and finally present it in a map together with images taken from Flickr.</div><div><br /></div><div>If you want to make an observation appear on the app just tweet something like:</div><div><br /></div><div><span class="Apple-style-span" style=" color: rgb(51, 51, 51); line-height: 15px; font-family:'Lucida Grande', sans-serif;"><i><span class="Apple-style-span" style="font-size:small;">#eBio observation: #Puma_concolor /-50.412673039931825, -100.713207244873047/ method:iPhonePhotoFlying Puma Concolor</span></i></span></div><div><br /></div><div>The idea is to mashup the data from Twitter together with the data from Flickr using the Darwin core machine tags. The reason for the second is that we are great supporters of people providing those tags and <a href="http://prezi.com/104535/view/#1">we created even some stickers to support them</a>.</div><div><br /></div><div>This is just an exercise to learn Flex 4 and promote the use of microformats and machine tags. But we hope you find it cool.</div><div><br /></div><div>Some comments. Magically the tweets from <a href="http://twitter.com/IvyMan">@IvyMan</a> on this disappeared while we were developing. Additionally there is not many machine tags in Flickr for the moment on darwin core terms, so there is some fakes over there too.</div><div><br /></div><div>All the credit goes to <a href="http://twitter.com/xavijam">@xavijam</a> for working on it! And from his experiments we are learning that is not going to be that easy to migrate to Flex 4.</div><div><br /></div><div style="text-align: center;"><span class="Apple-style-span" style="font-size:large;"><b><a href="http://vizzuality.com.s3.amazonaws.com/twitterClientEbio/TwitterClient.html">Open application</a></b></span></div><div><br /></div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com0tag:blogger.com,1999:blog-5286956948868856311.post-77868553917230332752009-06-08T13:49:00.010+02:002009-06-08T15:15:48.655+02:00World Database on Marine Protected Areas new website<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQxeID83IWcGfqGtBRdhb5lNN8Q3jF6-F7Eo-pEt5as00Egmt74xikFCaCp2vkgMU9rt-hFxYGBNJjJjREFnf8YhnoRqrV_gX9XnXHTM51SDWDBOXCGUzdQqlQailsmpzbrH4mqLg3ZdMe/s1600-h/logoSend.jpg"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 172px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQxeID83IWcGfqGtBRdhb5lNN8Q3jF6-F7Eo-pEt5as00Egmt74xikFCaCp2vkgMU9rt-hFxYGBNJjJjREFnf8YhnoRqrV_gX9XnXHTM51SDWDBOXCGUzdQqlQailsmpzbrH4mqLg3ZdMe/s320/logoSend.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5344943898480320930" /></a>The UNEP-World Conservation Monitoring Centre (UNEP-WCMC) today unveiled <a href="http://www.wdpa-marine.org/">The World Database on Marine Protected Areas</a> - a site designed to provide the most comprehensive set of Marine Protected Areas (MPAs) available.<div><br /></div><div style="text-align: center;"><span class="Apple-style-span" style="font-weight: bold;"><a href="http://www.wdpa-marine.org/"><span class="Apple-style-span" style="font-size:large;">http://www.wdpa-marine.org</span></a></span></div><div><br /></div><div>"With less than one percent of the oceans under legal protection, i</div><div>t is essential to maintain a dataset that focuses on MPAs and representation of the diverse species and habitats found in the marine environment." is read on the website.</div><div><div><div><br /></div><div><a href="http://www.vizzuality.com/">Vizzuality</a> has developed the User Interface and general design of the website, including the logo. Working together with UNEP-WCMC, and specially Craig Mills, we have developed innovative solutions to display this huge amount of data in a hopefully engaging website to invite people explore our oceans. </div><div><br /></div><div>But enough of "official words", here at biodivertido we would like to explain what are the technologies behind it and how things work under the hood. </div><div><br /></div><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 180px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSo3qNxB5V5Qo1NeOUd1qJbTxclLwygEEhLq6l5H7e9OVlsulRsxLLHIzWHfX4o_Lp655cQDypO7JD8_KrVBj0E54il8JsPq6VsAYtXLVzmQRUAohVD1MzqG4y2t2NohJ2o4jcJVmtk_sJ/s320/Picture+4.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5344931222235017746" /><div><span class="Apple-style-span" style="font-weight: bold;">Technologies:</span></div><div><br /></div><div>The website is a mix between HTML and Flash. The Flash application in the front has been developed using the <a href="http://en.wikipedia.org/wiki/Adobe_Flex">Flex framework</a> plus some Flash little things. On the ser</div><div>ver s</div><div>ide there is ASP.NET and <a href="http://www.themidnightcoders.com/products/weborb-for-net/overview.html">WebORB</a> to do <a href="http://en.wikipedia.org/wiki/Action_Message_Format">AMF remoting</a>. </div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">GIS:</span></div><div>The GIS engine behind the scenes is <a href="http://www.esri.com/software/arcgis/arcgisserver/index.html">ESRI ArcGIS Server.</a> WCMC pr</div><div>epared the different tiles</div><div> and caches for all layers. There is some places where we have used the new A<a href="http://resources.esri.com/help/9.3/ArcGISServer/apis/rest/index.html">rGIS Server REST API</a>.</div></div><div><br /></div><div>On the client side the whole project is very much based on the great <a href="http://code.google.com/apis/maps/documentation/flash/">Google Maps API for Flash</a>. We want to thank Pamela Fox from Google for her great support of the community. There is several techniques that we have introduced while working on this project like:</div><div><ul><li><a href="http://code.google.com/apis/maps/documentation/flash/demogallery.html?searchquery=Tile%20MouseOver&classname=">Tile Mouse Over</a>: To change the cursor when hovering over features on tiles.</li><li><a href="http://code.google.com/apis/maps/documentation/flash/demogallery.html?searchquery=WMS&classname=">WMS overlays</a>: Dynamically changing the Tile Overlays based on zoom levels for cached and not cached tiles.</li><li>Panoramio and Wikipedia markers without proxies.</li><li>Encoded Polilynes for multipolygons with inner rings.</li></ul><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 154px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV_wrAA4_rQ3DkQWITitGjfBzjZp-LzWXifA84faYcfePiRkCfB1WSLmoEGmGxenhV_NAon_wdj4t3lKgdD5-vJmLtfDObrbFx_UFeKbWQBknr-TzCHpC_RAfwYioOlJFZYDM4c_SqHfa4/s320/Picture+5.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5344941036002942610" /><div><span class="Apple-style-span" style="font-weight: bold;">Database:</span></div><div>WCMC is using <a href="http://www.microsoft.com/sqlserver/2008/en/us/default.aspx">Microsoft SQL Server 2008 Spatial Database</a> and we worked with them to generated Google Encoded Polylines out of the database. </div><div>There are also some things coming from the ESRI spatial database and some stats where dones using it, but I think the general idea is to mo</div><div>ve everything to SQL Server.</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">Deployment:</span></div><div>The whole website is being served from an <a href="http://aws.amazon.com/ec2/">Amazon EC2 instance</a>. The idea is also to make use of the A<a href="http://aws.amazon.com/cloudfront/">mazon Cloud Front CDN</a> to distribute tiles and other static files, but for the moment everything is in EC2. </div><div><br /></div><div>We will probably post more specific details on different parts of this project in the next weeks, but we wanted to give you a broad overview of how the project works and the different technologies being used.</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">One more thing...</span></div><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 155px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLKeaoIy-mcUTQ0_uyy9fcwVd45RHlnp7TkWXAyHj24J-SZm-iQlVgZHE-3Mg-uwsEaLKLWrrSGAOZlQowoj0jdAGeBGaeTsyWzNtqbix83R7fqcRBpOo7EoMppA7em_eOMXLNyK4MJYJB/s320/Picture+6.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5344941588388917218" /><div>One great thing of working with UNEP-WCMC is that all the source code that we have developed is Open Source! We are still working on specific details on what license the source will get, but for those curious, starting from today you can checkout all source code from <a href="http://code.google.com/p/vizzuality/source/browse/#svn/trunk/MapWidgetBase">Vizzuality SVN repository</a>. Please let us know if you find it interesting!</div><div><br /></div><div>We are still under Beta phase of the project, so it might be that you find some bugs, <a href="http://code.google.com/p/vizzuality/issues/entry">please report them</a>!.</div><div><br /></div><div>Thanks again to UNEP-WCMC to let us work on this great project and look forward to make it better!</div></div><div><br /></div><div><br /></div></div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com1tag:blogger.com,1999:blog-5286956948868856311.post-33268473554177067752009-06-06T12:02:00.004+02:002009-06-06T13:01:00.928+02:00Using Google Spreadsheets with Google Maps for FlashI have recently been working on a little project that had a very simple purpose: Put in a map around 200 markers about where to find dog waste bags disposals in a town. This might have little intersection with biodiversity, but I think some of the ideas might be useful for other people.<div><br /></div><div>The idea was to create a simple map/widget that could be managed by a non technical person and did not require setting up databases and hosting services. The kind of project you want to set up and forget more or less about it.</div><div><br /></div><div>Well, the simplest thing to manage the location of the disposal is something like and spreadsheet and the internet version for that is <a href="http://docs.google.com/">Google Spreadsheets</a>. I started thinking on using the <a href="http://earth.google.com/outreach/tutorial_spreadsheet.html">Spreadsheet mapper</a>, but it has far too much options, and we did not really need to share a KML file. So I thought creating something simpler. Just my required columns on the document and connect from <a href="http://code.google.com/apis/maps/documentation/flash/">Google Maps for Flash</a>. So we wo</div><div>uld be able to just distribute the Flash SWF file and it will take </div><div>care of connecting to Google Spreasheets, download the data and display it on the map.</div><div><br /></div><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 249px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFyC-SXu8a-7pBY_4vhAO1bPfKxY_CQIZ6tHcYZsMHMkNb8NilrLmf5_EO0rPKPMb__OjV_X-g1fkcqukndLmcDstsQspsc2XSEDLQFmGzDMOzwa1yT1HeqAdV3Q0P70eFg4h1oHeZxwHu/s320/Picture+3.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5344160504377063858" /><div>Sounds very easy no? Just publish the spreadsheet, setting automatically republish on changes, select the CSV format, and you get a perfect API to your data. <a href="http://spreadsheets.google.com/pub?key=r1nD0NusVHyIyRKVbDoMX2A&single=true&gid=1&output=csv">Look at ours for example</a>.</div><div>Then on the Flex app is as simple as parsing a CSV and dynamically create the needed markers.</div><div><br /></div><div>The only problem is... Google does not like crossdomain.xml f</div><div>iles like <a href="http://twitter.com/crossdomain.xml">everybody</a> <a href="http://api.flickr.com/crossdomain.xml">else</a> <a href="http://www.amazon.com/crossdomain.xml">in</a> <a href="http://profile.ak.facebook.com/crossdomain.xml">the</a> <a href="http://data.gbif.org/crossdomain.xml">2.0 internet</a>. </div><div><br /></div><div>Therefore we were gonna have to create a proxy server to bypass the <a href="http://riaon.com/crossdomain-policy-flex/">security restrictions</a>, but we really did not like the idea of havi</div><div>ng to set up one just for this small thing and maintaining it. </div><div><br /></div><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 320px; height: 192px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirMIp-K-_2hvmnsqILmiM1SwnzTa0K0Do4NNgRhkk25NB1SPlkQvtSMCSUuwF4xXg4T6y0M_lWW-aFYy5-bWv6VnN8Q9DqEkaOph1zRA3AWe_nVXtvB691uD8c1wxhJF3eudtuduLixnig/s320/Picture+4.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5344164670209154370" /><div>So I decided to take a look at <a href="http://pipes.yahoo.com/">Yahoo Pipes</a> to <a href="http://blog.satishkore.com/?p=7">bypass the crossdomain issue</a>. You just need to create a simple Pipe that consumes the CSV from spreadhseets and output it as JSON. Yahoo pipes has an open crossdomain file, so no problems. <a href="http://pipes.yahoo.com/jatorre/sanecan">Here is my pipe</a> for example. Very simple and effective.</div><div><br /></div><div>You can see the final result in <a href="http://vizzuality.com.s3.amazonaws.com/sanecan/SANECANmap.html">here</a>. And of course you can always grab the code from <a href="http://code.google.com/p/vizzuality/source/browse/trunk/SANECANmap/src/SANECANmap.mxml">Vizzuality Google Code repository</a>. </div><div><br /></div><div>The project had to deal also with transforming UTMs into Latitude/Longitudes and some other issues, but I think this overview is enough. The source code is so simple that I dont think its needs more explanations.</div><div><br /></div><div>We would like to start using the Google Data API much more in the future, specially the shiny new <a href="http://code.google.com/apis/maps/documentation/mapsdata/">Google Maps Data API</a>, but I think this makes for a very simple solutions for lot of small projects like this one.</div><div><br /></div><div>And finally, Google, please start setting up crossdomain files on your APIs, or at least, explain us why you dont do so... the actual situation is very frustrating.</div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com1tag:blogger.com,1999:blog-5286956948868856311.post-17423228270259276252009-06-04T17:46:00.002+02:002009-06-04T17:52:45.978+02:00Greenpeace BlackPixel | Beautiful Idea<div><br /></div><div>Today we've discovered <a href="http://www.greenpeaceblackpixel.org/#/en/">BlackPixel</a>. A beautiful initiative by greenpeace to save energy.</div><div>The application draws a black square in your screen that not disturb your visualization.</div><div>If you over the square, you can view how many energy is being saved.</div><div><br /></div><div>A little effort for you, a big benefit for all.</div><div><br /></div><div>You can know more about it <a href="http://www.greenpeaceblackpixel.org/#/en/">here</a></div>Sergio Alvarezhttp://www.blogger.com/profile/02466838358242522224noreply@blogger.com4tag:blogger.com,1999:blog-5286956948868856311.post-34657753564509803002009-05-29T08:58:00.008+02:002009-05-29T10:45:22.058+02:00We are going to E-biosphere 09<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOuo4-5C9HLIp3MJr9jlJfgq132-EdZWIL-jFzQJWl5n_7HnljzIrWvkw6LY6TQtQ0KPCpwrPnTGKXPeRAPwTHIIcW43oG9GyqI9SDojN07FRYSGg1yMTwFCRZtFWbvhYALghgjrHGOQRK/s1600-h/Picture+1.png"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 266px; height: 90px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOuo4-5C9HLIp3MJr9jlJfgq132-EdZWIL-jFzQJWl5n_7HnljzIrWvkw6LY6TQtQ0KPCpwrPnTGKXPeRAPwTHIIcW43oG9GyqI9SDojN07FRYSGg1yMTwFCRZtFWbvhYALghgjrHGOQRK/s320/Picture+1.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5341160846886769858" /></a><a href="http://www.e-biosphere09.org/">E-Biosphere 09</a> starts next Monday 1st of June on London. Today we want to announce that most of us, Tim Robertson, Sergio Alvarez, Dave Martin and Javier de la Torre, will be there. <div><br /></div><div>Even more, we have a booth! <a href="http://www.gbif.org/">GBIF</a> & <a href="http://www.vizzuality.com/">Vizzuality</a> are joining efforts to showcase the recent projects we have been working on and some of the future ones. We are all very excited to showcase what kind of things can be done with biodiversity data and visualizations. <div><br /></div><div>Tim, Sergio and Javier will be present at the booth, but we would like to be present online at the same time. Our idea is to use this blog as the online platform to present the different ideas, prototypes, wireframes, etc that we will showcase at E-biosphere 09. Especially as Andrew could not join us for London. So the blog is our online booth!<br /></div><div><br /></div><div>In the next few days we will be working hard on different topics, including species distribution, analyses, data integration, indexing, visualisations, taxonomy, data publishing, computing on the cloud, migration paths etc. So stay tuned to know what is going on in multiple posts that we will publish here.</div><div><br /></div><div>Aditionally you can follow some of us on Twitter:</div><div><a href="http://twitter.com/jatorre">@jatorre</a></div><div><a href="http://twitter.com/saleiva">@saleiva</a></div><div><a href="http://twitter.com/andrewxhill">@andrewxhill</a></div><div><a href="http://twitter.com/timrobertson100">@timrobertson100</a></div><div><br /></div><div>If you happen to attend e-biosphere please come and join us at the booth, if not we look forward to get your comments on the blog.</div><div><br /></div><div>See you all in London!</div><div><br /></div><div>Javier, Tim, Sergio, Dave, Andrew.</div></div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com1tag:blogger.com,1999:blog-5286956948868856311.post-58168342456023914092009-05-29T07:23:00.005+02:002009-05-29T07:56:54.186+02:00A new contributor to BiodivertidoSo, I’ve decided to become a blogger. But Javier said that on my quest to riches, I needed to first give a quick introduction. Here you go.<br /><br />My name is Andrew Hill. I am working on my PhD in <a href="http://robgur.googlepages.com/">Rob Guralnick’s</a> lab in the Ecology and Evolutionary Biology department at the University of Colorado. I began working here almost 3 years ago on a project to reconstruct the evolution and spread of influenza through time and over geography (see <a href="http://www.ogleearth.com/2007/04/mapping_the_phy.html">here</a> or <a href="http://dx.doi.org/10.1080/10635150701266848">here</a>). Like the influenza itself, my interests have mutated rapidly. Most of my work now involves at least one of three informatics subtypes; biodiversity informatics, phyloinformatics, or (I’m not sure people say this) visual informatics. I have ongoing research on the evolution of influenza, as well as a handful of smaller genomics and bioinformatics projects, and am just now starting work for the <a href="http://www.nbii.gov/portal/community/Communities/Habitats/Marine/Marine_Data_%28OBIS-USA%29/">OBIS-USA</a> project.<br /><br />As for my role at Biodivertido, I don’t know yet. Rob, <a href="http://www-biology.ucsd.edu/labs/jetz/">Walter Jetz</a>, and <a href="http://eol.org/">EOL</a> just put on an excellent workshop in Chicago that gave me, Tim, and Javier the opportunity to talk about our interests and coordinate some of our efforts on a couple of new projects. We will likely be giving you more details about some of those in a few hours. So don’t go far from your RSS feed.<br /><br />For now, all I can say is that I’m extremely excited to be working with these guys and will hopefully be contributing cool things very soon.Andrew W Hillhttp://www.blogger.com/profile/04814436364156670727noreply@blogger.com2tag:blogger.com,1999:blog-5286956948868856311.post-54341709980642189912009-04-29T01:18:00.002+02:002009-04-29T01:31:25.969+02:00GBIF-World Database on Protected Areas project<div>The GBIF-WDPA project intersects together the data from GBIF and the World Database on Protected Areas. It was developed by Vizzuality for GBIF and IUCN. I have talked about this project in different posts already, but I thought it could be interesting to share an screencast I created some time ago about it.</div><div><br /></div><div>The video has a little introduction on what was the project about, what technologies were used and finally a demo of the application itself.</div><div><br /><object width="500" height="281"><param name="allowfullscreen" value="true"><param name="allowscriptaccess" value="always"><param name="movie" value="http://vimeo.com/moogaloop.swf?clip_id=3284706&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=00ADEF&fullscreen=1"><embed src="http://vimeo.com/moogaloop.swf?clip_id=3284706&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=00ADEF&fullscreen=1" type="application/x-shockwave-flash" allowfullscreen="true" allowscriptaccess="always" width="500" height="281"></embed></object><br /></div><div><br /></div><div>Ah! If you want to try the tool for yourself it is finally published. Try it for:</div><div><br /></div><div><a href="http://widgets.gbif.org/pa/#country/au">Australia</a></div><div><a href="http://widgets.gbif.org/pa/#country/us">United States</a></div><div><a href="http://widgets.gbif.org/pa/#country/es">Spain</a></div><div><a href="http://widgets.gbif.org/pa/#country/de">Germany</a></div><div><br /></div><div><a href="http://widgets.gbif.org/pa/#/area/2571">Great Barrier Reef</a></div><div><a href="http://widgets.gbif.org/pa/#/area/975">Yosemite</a></div><div><a href="http://widgets.gbif.org/pa/#/area/151242">Posets Maldeta (Pyrenees)</a></div><div><br /></div><div>We are working right now on some more advanced visualizations, but this will come hopefully soon. I hope you enjoy it.</div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com0tag:blogger.com,1999:blog-5286956948868856311.post-55841862072598965122009-04-14T10:00:00.035+02:002009-04-24T02:32:27.988+02:00WMS Tiling Clients<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgToyu1a0730q4YQo1liv1MHDbBbO8SkLgHSoFSzT92b3WdnMMjyOCz4i3npSUY-8ZtBdCl79efjHY_4SyNZvNDnQsU0oqvQdRg8jRVMqd-E0pzjATghV7Fig1MwVBjXpKCk1yAX-4kw2c/s1600-h/screenshot.jpg"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 200px; height: 118px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgToyu1a0730q4YQo1liv1MHDbBbO8SkLgHSoFSzT92b3WdnMMjyOCz4i3npSUY-8ZtBdCl79efjHY_4SyNZvNDnQsU0oqvQdRg8jRVMqd-E0pzjATghV7Fig1MwVBjXpKCk1yAX-4kw2c/s200/screenshot.jpg" alt="" id="BLOGGER_PHOTO_ID_5324462273504366850" border="0" /></a>For projects relying on open source software and using <a href="http://en.wikipedia.org/wiki/Web_Map_Service">WMS</a> services for their map applications perhaps the most obvious choice for a client application is to write something using the OpenLayers API. This is a very extensive javascript API and is used to demonstrate the open source OGC compliant servers <a href="http://mapserver.org/">MapServer</a> and <a href="http://geoserver.org/">GeoServer</a>. Its straight forward to construct a map and then to overlay multiple layers using WMS services.<br /><br />The screenshot above shows a layer of 0.1 degree cell densities representing data from Australian National Herbarium rendered using GeoServer layered on top of the Google satellite base layer, all pulled together using OpenLayers.<br /><br />Running this locally the map was rendered by OpenLayers rather quickly. Running this on an externally hosted server I began to notice curious loading of tiles by OpenLayers. So I did a stripped down comparison of the same functionality using the Google maps API. Rendering the cell density tiles from a WMS service in Google maps was done using a javascript function written by <a href="http://johndeck.blogspot.com/">John Deck</a> - available <a href="http://chignik.berkeley.edu/google/wms236.js">here</a>.<br /><br />The performance improvement with Google maps was immediately obvious. Using the Firefox <a href="http://developer.yahoo.com/yslow/">YSlow</a> plugin we see:<br /><table style="margin: 0pt; padding: 0pt;"><br /><tbody><tr><td style="text-align: center;">OpenLayers</td><td style="text-align: center;">Google Maps API</td></tr><br /><tr><td><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja-dWV_xNRQReWCyesFYLO0fSyKJV-HpgcwNz_2DiCzRgO_5IsDTaupoYpFnzdZKAHmuuz_88Ubk6pXtfomDq1prJF3VUBN_kPjTiRSMCJ92BCVkaDk5BcW9njPTebtyGEB2R00cqT6qM/s1600-h/openlayers-result.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 254px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja-dWV_xNRQReWCyesFYLO0fSyKJV-HpgcwNz_2DiCzRgO_5IsDTaupoYpFnzdZKAHmuuz_88Ubk6pXtfomDq1prJF3VUBN_kPjTiRSMCJ92BCVkaDk5BcW9njPTebtyGEB2R00cqT6qM/s320/openlayers-result.png" alt="" id="BLOGGER_PHOTO_ID_5324473994850313458" border="0" /></a><br /></td><br /><td><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJDplH9Zxwm6INEcv2ecCsaDuOCgB2qlOF8TNkVRoml0jAqlSPA2fg9QA1PTgQCqDr_PCUBR20H7MkA-KBu68kVfWXlvTYuyFu0jU9c34bxSw_G3dLcGCjJ0whWu5LNjWIogWJs_U4x1Q/s1600-h/google-result.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 265px; height: 320px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJDplH9Zxwm6INEcv2ecCsaDuOCgB2qlOF8TNkVRoml0jAqlSPA2fg9QA1PTgQCqDr_PCUBR20H7MkA-KBu68kVfWXlvTYuyFu0jU9c34bxSw_G3dLcGCjJ0whWu5LNjWIogWJs_U4x1Q/s320/google-result.png" alt="" id="BLOGGER_PHOTO_ID_5324474103399089698" border="0" /></a><br /></td></tr></tbody></table>So the main thing of interest here is the number of tiles loaded by OpenLayers compared to Google maps (91 vs 50 - although oddly the total size of the images loaded is roughly equivalent, and the tile size for both seems to be 256x256 pixels). Increasing the size of the tile for OpenLayers does reduce the number of tiles requested (I tested with 512x512 and 1024x1024) but this has little effect on performance in openlayers.<br /><br />So the performance difference between Google Maps and OpenLayers could be accounted for by any combination of the following:<br /><ul><li>Incorrect use of the API (test for OpenLayers is <a href="http://ala-portal.googlecode.com/svn/trunk/ala-portal-web/gis/openlayers.html">here</a>) and we are missing configuration to reduce tile loading in openlayers<br /></li><li>The tile loading algorithm for Google maps is more efficient in only loading the required tiles for selected view areas</li><li>Openlayers is preloading more images to speed up panning. This would be good, but OpenLayers doesnt seem to prioritise the loading of currently viewed tiles.<br /></li></ul>The source for the test for OpenLayers is <a href="http://ala-portal.googlecode.com/svn/trunk/ala-portal-web/gis/openlayers.html">here</a>, and for Google maps <a href="http://ala-portal.googlecode.com/svn/trunk/ala-portal-web/gis/googlemaps.html">here</a>.Dave Martinhttp://www.blogger.com/profile/15236961259181793280noreply@blogger.com2tag:blogger.com,1999:blog-5286956948868856311.post-18144221697819288882009-04-02T16:20:00.027+02:002009-04-03T00:56:02.469+02:00The Taxonomy browser visualization #1 - Tree Lists<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_I-1-u0k_5aA/SdUy7sN01yI/AAAAAAAAABs/kqdIJ3m2s_U/s1600-h/treelist_w.jpg"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 230px; height: 320px;" src="http://2.bp.blogspot.com/_I-1-u0k_5aA/SdUy7sN01yI/AAAAAAAAABs/kqdIJ3m2s_U/s400/treelist_w.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5320214535870928674" /></a>Hi, my name is <a href="http://www.linkedin.com/in/saleiva">Sergio Alvarez Leiva</a> and this is my first pos at biodivertido, Finally. I'm part of Vizzuality and work as Interaction Designer for GBIF among others. I will mainly be posting about Interaction Design, UX and Design in general, althought I'll try to post about Front.end developing a little. I hope you find it interesting.<br /><div><br />Since I began designing for Biodiversity data, I've encountered lot of interesting challenges related to the size of the datasets .<div>Maybe, <span class="Apple-style-span" style="font-weight: bold;">one of the hardest challenges is designing a taxonomy browser visualization.</span> The taxonomy browser can represent up to 1.7M names, with nodes with more than 200 childs. That makes it complicate in terms of interaction design and performance.</div><div>I will write in a series of post my impressions on different techniques we have been experimenting:</div><div><br /></div><div>In this first post I'm going to talk about tree list visualizations, but before I get into details, I'd like to talk a little about generic concepts that might be applied to all the visualizations. There is a lot of bibliography about trees out there, but let me introduce here my own "easy" concepts.</div><div><br /></div><div>When browsing a tree you need to know where you are and what is around you, thats parent and brothers and in general this is called <span class="Apple-style-span" style="font-weight: bold;">Contextualization</span>. Then you need to be able to find the childs, thats <span class="Apple-style-span" style="font-weight: bold;">Discovering</span>. All this needs to be easy to do, <span class="Apple-style-span" style="font-weight: bold;">Usability and UX,</span> and finally must be easy to integrate with the application, <span class="Apple-style-span" style="font-weight: bold;">Easy integration.</span></div><div><br /></div><div>On those terms, the tree list visualization presents some <span class="Apple-style-span" style="font-weight: bold;">advantages and disadvantages.</span></div><div><br /></div><div><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: bold;"> </span></span><span class="Apple-style-span" style="font-weight: bold;">- Contextualization:</span> Maybe the list visualization is the most standard way to solve this problem. The tree lists lets the user discover the whole tree by clicking in the different nodes. But the problem is the vertical size that the tree gets when the user deployes a node (This is an integration problem too). When the user will have deployed more than 3 levels, he will have to use the vertical scroll, and probably, this action will make the contextual information (parent nodes and "brothers") out of the visible screen area.</div><div><br /></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>This problem could be solved implementing some variations in the Tree list; We could hide all the nodes not related to the selected child and get more space for view only the important nodes.<br /></div><div><br /></div><div><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: bold;"> </span></span><span class="Apple-style-span" style="font-weight: bold;">-Poor UX : </span>"Discovering biodiversity". We have to involve the users in our play and invite they to discover more and more. I think the Tree Lists are a little boring...we need more action!<br /></div><div><br /></div><div><span class="Apple-tab-span" style="white-space:pre"><span class="Apple-style-span" style="font-weight: bold;"> </span></span><span class="Apple-style-span" style="font-weight: bold;">- Usability:</span> This visualization is pretty used and the users will understand how it works quickly. On the other hand, I think that in a Tree list is not so easy to find the desired node - the users would have to deploy a lot of nodes, and scroll a lot too, before find their objective. Maybe would be positive for this point to implement a search box. It might depend of the data size.</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">Conclusion; </span>We've a standard visualization that lets us to implement a Taxonomic browser in a lot of different situations. I think this is not the best solution that we could implement, but its true that we could use this always. </div><div> <br /></div></div>Sergio Alvarezhttp://www.blogger.com/profile/02466838358242522224noreply@blogger.com5tag:blogger.com,1999:blog-5286956948868856311.post-53502444074040934452009-03-30T23:09:00.006+02:002009-03-31T09:19:26.923+02:00SpatialKey and biodiversity primary data analysis<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJpsdsaFWMfCXwQmeuF0Fp7KMFjUL_bO8URuuBfZG8vvCAo6I8y2eu296LhSlBk1nt-4TpeAeiXx7fIIvpPi9kk8GyIp1dz9ZvNJkOQoac-KvtQfBNgHvFx5k8PXjcifeazo_Yuitr8Bs1/s1600-h/Picture+4.png"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 198px; height: 86px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJpsdsaFWMfCXwQmeuF0Fp7KMFjUL_bO8URuuBfZG8vvCAo6I8y2eu296LhSlBk1nt-4TpeAeiXx7fIIvpPi9kk8GyIp1dz9ZvNJkOQoac-KvtQfBNgHvFx5k8PXjcifeazo_Yuitr8Bs1/s320/Picture+4.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5319092037988330642" /></a><br /><div>Some days ago the good people from <a href="http://universalmind.com/">Universal Mind</a> open the beta program for their new product called <a href="http://www.spatialkey.com/">SpatialKey</a>. For those who are not in the Flex community, Universal Mind is a very recognized company developing Rich Internet Applications. So it was a great pleasure when I saw<a href="http://biodivertido.blogspot.com/2008/08/gbif-data-heat-maps-heat-maps-over.html"> months ago</a> that they were working on a new geospatial product for data analysis.</div><div><br /></div><div>I was lucky to get an invitation to the beta program and be able to take a look. They are promising some great things, but for the moment the beta is limited in certain ways I will describe later. The best to get an overview of it is to watch some of their ubercool videos (maybe too much for my taste). </div><div><br /><object width="480" height="295"><param name="movie" value="http://www.youtube.com/v/7UJVAAg53kY&hl=en&fs=1"><param name="allowFullScreen" value="true"><param name="allowscriptaccess" value="always"><embed src="http://www.youtube.com/v/7UJVAAg53kY&hl=en&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="480" height="295"></embed></object><br /></div><div><br /></div><div>I wanted to give it a try as soon as possible, and coincidentally I just finished working on the new WDPA-GBIF widget. The widget allows you to visualize biodiversity primary data, from GBIF, for all protected areas in the world. Check out for example the <a href="http://widgets.gbif.org/test/PACountry.html#/country/AU">protected areas in Australia</a>. Then select for example an area like the<a href="http://widgets.gbif.org/test/PACountry.html#/area/1332"> Great Barrier Reef.</a> You will be able to download the data in multiple formats.</div><div><br /></div><div>With this I downloaded the data for the Canadian Rocky Mountains and imported it in SpatialKey. The import now is limited to 10.000 records and 25 columns, so I had to delete multiple columns and records. I think SpatialKey should allow the discard of the data visually when it is already uploaded.</div><div><br /></div><div>With SpatialKey you manage separately your datasets and the reports you create based on them. The reports right now are not exportable to the outside. I mean, you can not print or distribute as a widget the report you have created. I know they are working on it, but for the time being I only have 2 possibilities to show you how it looks like: share my report with everybody that gives me his email, or just do a little screencast for everybody to watch here. For the first one if someone is really interested to get into the system to take a look, send me an email. the screencast is following:</div><div><br /></div><div><object width="500" height="350"><param name="allowfullscreen" value="true"><param name="allowscriptaccess" value="always"><param name="movie" value="http://vimeo.com/moogaloop.swf?clip_id=3935750&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=00ADEF&fullscreen=1"><embed src="http://vimeo.com/moogaloop.swf?clip_id=3935750&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=00ADEF&fullscreen=1" type="application/x-shockwave-flash" allowfullscreen="true" allowscriptaccess="always" width="500" height="350"></embed></object><br /></div><div><br /></div><div>So here goes the things that I like a lot:</div><div><ul><li>The heatmaps are just gorgeous. I would love to know how they do it.</li><li>The timeline filter is great. Has some usability issues but is great.</li><li>The way grids are displayed for summaries. The hover effect is very good and the tooltip very clear.</li><li>The filter "pods" are nice, but I wonder what would happen when you have thousands of hundreds records to search or select on. I suppose that when there is lot of data only the search would be enabled and not the selection.</li><li>Great look and feel.</li></ul><div>Other comments following:</div><div><ul><li>Is it necessary to refresh on every map movement? I understand it is on the zoom and if you have the filter by visible area disabled.</li><li>Not having the possibility right now to share the reports as widgets to embed on the blog.</li><li>It would be nice to also let the user provide a polygon or geometry to define the boundaries of the analysis. In this case for example would help a lot to visualize the borders of the protected area.</li></ul><div>And finally, the things I really wonder how they work internally:</div><div><ul><li>The heatmaps!</li><li>The data structures they use for dynamically regrouping the data on the client.</li><li>If it is true that they can handle millions of records, how does the server infrastructure looks like. I know it is Java, but what about the data store, how can they handle the creation of dynamic indexes or how do they do it? Would it work with GBIF data?</li></ul><div>My general impression of the tool is great. It looks awesome and works really well. It looks very similar to some of the ideas we have for developing analytical tools for biodiversity data with GBIF. Tim give your impressions please!</div></div></div></div><div><br /></div><div>I would love to see more and more such analytical tools for biodiversity. How they call them? Something like Bussiness Intelligence, I think we need some of this in our community. For the time being I will try to get into talks with Universal Mind on the applicability of SpatialKey for huge biodiversity primary data like GBIF. </div><div><br /></div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com3tag:blogger.com,1999:blog-5286956948868856311.post-79560945152705689212009-03-20T17:38:00.009+01:002009-05-23T16:14:43.283+02:00How many zoom levels are enough?While processing the GBIF data index for all species to display the maps shown in the <a href="http://biodivertido.blogspot.com/2009/02/grid-data-shared-as-point-data-errors.html">last post</a> I though it worth showing the number of "occurrences per species per cell" at the various zoom levels.<br />We make use of the tiling mechanism employed by many mapping clients, who request 256x256 pixel tiles and then we process the data to be several zoom levels ahead of the one displayed. It is really quite simple, and best described with a couple of examples. <div><br /></div><div>Processing to 4 zoom levels ahead looks like:<div><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbysNkQ5q9XADOIr-vfUhJLetrUeFGDDgRMpQ4UZckjTbjSlrHl25AVEDrc0bbi1ANbrc_llnSIOKj3mgBYrdGqxoeJ9gNYC2EPUaPdg4CHFVMlBUaJWwpwj9H2CS6TBuVPDKLhJtLOQ/s1600-h/zoom2.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 252px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbysNkQ5q9XADOIr-vfUhJLetrUeFGDDgRMpQ4UZckjTbjSlrHl25AVEDrc0bbi1ANbrc_llnSIOKj3mgBYrdGqxoeJ9gNYC2EPUaPdg4CHFVMlBUaJWwpwj9H2CS6TBuVPDKLhJtLOQ/s320/zoom2.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5315315485109114530" /></a><br /><br /></div><div><br /></div><div>Processing to 6 zoom levels ahead looks like:</div><div><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTb_QcLe7XTHumlur6N0AtBRn5XnbeqTKW_SjAwNVN95c8qmGGZ0pkIim_UR4Xhlo1t3W8nNk6w9W9_FQygjPeiS5zFjBLf1ZdzzKGaUit2qNaSD9JL02zM12FyX8lP_2RequNjkY4gQ/s1600-h/zoom4.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 225px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTb_QcLe7XTHumlur6N0AtBRn5XnbeqTKW_SjAwNVN95c8qmGGZ0pkIim_UR4Xhlo1t3W8nNk6w9W9_FQygjPeiS5zFjBLf1ZdzzKGaUit2qNaSD9JL02zM12FyX8lP_2RequNjkY4gQ/s320/zoom4.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5315315567461158562" /></a><br /></div><div><br /></div><div>When processing to 6 zoom levels, the following shows where it becomes unnecessary to process anymore (around zoom level 11):<br /></div><div><br /></div><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRuH6RVzvDTBSgarXndP9jFFjoDJMAP4hX6EtXplyb6qEeX9wsqAsaYGD91zJEj1IwzqsbopkMcnh3Dvgse1O0y2WtYBBGODCKQIdyPLO8_2b3RJM8pd_8noL3fJiF3MnrdPZlthaU1w/s1600-h/graph.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 233px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRuH6RVzvDTBSgarXndP9jFFjoDJMAP4hX6EtXplyb6qEeX9wsqAsaYGD91zJEj1IwzqsbopkMcnh3Dvgse1O0y2WtYBBGODCKQIdyPLO8_2b3RJM8pd_8noL3fJiF3MnrdPZlthaU1w/s400/graph.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5315316983027128258" /></a></div>Tim Robertsonhttp://www.blogger.com/profile/07889700598656669041noreply@blogger.com0tag:blogger.com,1999:blog-5286956948868856311.post-27467482164509746962009-02-24T10:49:00.011+01:002009-03-09T15:41:11.439+01:00Grid data shared as point data. Errors and visualization problems<div style="text-align: justify;">In todays world the easiest way to share location is using Decimal latitude and longitudes, and preferably on <a href="http://en.wikipedia.org/wiki/World_Geodetic_System">WGS84</a>. With such coordinates you can make use of lot of existing data <a href="http://wiki.tdwg.org/twiki/bin/view/DarwinCore/WebHome">transfer standards</a>, <a href="http://en.wikipedia.org/wiki/Comparison_of_Web_Map_Services">mapping APIs</a>, or <a href="http://openmodeller.sourceforge.net/">analysis tools</a>. This is good because it helps a lot on interoperability and let developers and scientist easily mix data and use it together.<br /></div><div style="text-align: justify;"><br /></div><div>But, most of biodiversity primary data, the location of a species at a certain place in a certain moment, was taken before GPS and even after lot of different coordinate systems were used. For example <a href="http://en.wikipedia.org/wiki/Universal_Transverse_Mercator_coordinate_system">UTM</a>. In those coordinates systems people do not indicate an exact position, like you do with lat/lon, but an area or zone. That is ok for most uses, you dont need to know the exact position of where a specimen was collected, and sometimes it is even much easier to use those zones, or cells or area, to collect and aggregate data.</div><div style="text-align: center;"><br /></div><div>The problem comes when you start sharing your data in public networks like GBIF. Most of data providers in the GBIF network, if not all, provide their data using lat/long. This is even the recommended method to easier process and aggregate the data. Therefore what most providers are doing is transforming those areas into points by taking the centre of the area.</div><div>If you put this transformed coordinates with other real lat/long coordinates, like GBIF does, then you end up not knowing what was originally a cell or an area and what was really a point.</div><div style="text-align: center;"><br /></div><div>This shows the result when represented in a map (click to see it bigger). Or better<a href="http://www.imastedev.com/eol/EOLSpeciesMap.html?taxon_id=13839800"> use this application to browse the data for yourself</a> (development server).</div><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 242px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7hJoDm1J_2chVpob_Bilsx9hU4z8amBBzPd-3mlz-chy2rxVh623r0EJejvZ2dW1bYw66Ce2rRGdeyvIu7N6MOfpvY2izGhU6i3jvFJI6JeIiWz79jWy7IZLguIvDCE40Uzryh5UCF260/s400/spain_france_austria_example.jpg" border="0" alt="" id="BLOGGER_PHOTO_ID_5310026678100136034" /><div style="text-align: center;"><br /></div><div><div style="text-align: center;"><br /></div></div><div><br /></div><div>This is all the data GBIF has from different data providers about Passer domesticus (House sparrow). You can see that Spain, France and Austria seem to have some weird data. They look like if the a gridified. Specially if you compare it to the US:</div><div><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 320px; height: 214px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCECyiAqByeXN5uWvS49r03vvgNLNVlJvyHJHJTl6-IRelOebIowlrJ6HSrtDC9DHIuQOhATHEn9oYHzMnIj4QocX4UYjTUfWK0RUwsu3LF8ppwh5-BihkdOwakGGNYvM5EAUpYfnCDVi0/s320/us_example.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5310027953398892354" /><div style="text-align: center;"><br /></div></div><div style="text-align: center;"><br /></div><div style="text-align: left;">The reason is that most of the data in Spain, France and Austria (also Germany but is not shown here) is derived from UTM data or another form of "area" coordinates. </div><div>And the worst is that we can not know which data is actually grid data and what is actual points, we can only see it on a map like this.</div><div><br /></div><div>This has some repercussions:</div><div><br /></div><div><b>Errors:</b></div><div>We are introducing errors to the data. The user do not know the "resolution" or quality of the data coming from GBIF. For example in France, the data in GBIF says that there is a Passer domesticus at the red point, when actually it could be in any of the greed square. Thats 25Km error from one side to the other</div><div><br /></div><div><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 178px; height: 133px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7FWVqUoE4zO5__vtqcDQsYP4evmZ8B-6ecS3wb_BehJoxf64S71hp10fzW3-kDfTsmQKcBxYUFooKFMPmYH6orKBkB_IYDzmnZTt-IjqIUuP3s3Urv1xe2qgKf89FiMxVLpKMrHW7q8-u/s320/france_one_cell_example.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5310030986477418482" /></div><div style="text-align: left;">In Spain the error is around 10Km and in Austria is around 900m.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><b>Visualization:</b></div><div style="text-align: left;">Without knowing what is a point a what is an area, it is very complicate to do any visualization that does not look strange in some areas. Experience users will understand why it might be like this, but most of the users will not. And people will keep zooming into a point thinking they can get to see the exact position of where a species was observed or collected, when actually this point is not real at all, it is just a visualization error due to the underlaying data problem.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">Of course there is some works around. People are starting to share their coordinates with an error indicator. Thats good, but this is not an actual error from my point of view, it is just a different way of collecting locations, the error is in thinking that this is actually a point, when it is not, it is an area.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><b>Fitness for analytical use:</b></div><div style="text-align: left;">Consider modeling predictive species distribution based on known (point) occurrence data for areas having similar environmental conditions (Environment Niche Modeling). The results of any model would be wildly inaccurate considering a 25Km deviation of input data. Some indication is necessary so that data rounded to the nearest grid is not considered valid for this use. </div><div style="text-align: left;"><br /></div><div style="text-align: left;"><b>Possible solutions:</b></div><div style="text-align: left;">What possibilities we have? Well, I think the best would be to let the user share their data in the way they have it. Of course you can do that already in comments and things like that, but thats not very convenient.</div><div style="text-align: left;">So I think the best is that if people has location based on UTMs they share them like they are and also the Spatial Reference System as <a href="http://en.wikipedia.org/wiki/Well-known_text">Well Know Text</a>. So for example UTM10N would be:</div><div style="text-align: left;"><pre name="code" class="php">PROJCS["NAD_1983_UTM_Zone_10N",<br />GEOGCS["GCS_North_American_1983",<br />DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137,298.257222101]],<br />PRIMEM["Greenwich",0],UNIT["Degree",0.0174532925199433]],<br />PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],<br />PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-123.0],<br />PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_of_Origin",0.0],<br />UNIT["Meter",1.0]]</pre>and then they will have to share the <a href="http://en.wikipedia.org/wiki/Easting_and_northing">easting and northing</a>, like 630084m east, 4833438m north.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">With this information the end user can then decide if they want to transform into lat/long if necessary, but at least they know what they are doing.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">Additionally, indications that records have been rounded to a grid are required to determine their fitness for use.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">I dont have experience using UTMs so I might be wrong on some of the things I have said, but at least I hope you get an idea on what the issue is and why I think is important to work on it.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><b>Update:</b> I should have mention that the ABCD TDWG Standard actually supports more or less what I said by providing atomic concepts for sharing UTM data. These are:</div><div style="text-align: left;"><pre name="code" class="php"><br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesUTM<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesUTM/UTMZone<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesUTM/UTMEasting<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesUTM/UTMNorthing<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesUTM/UTMText<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesGrid<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesGrid/GridCellSystem<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesGrid/GridCellCode<br />/DataSets/DataSet/Units/Unit/Gathering/SiteCoordinateSets/SiteCoordinates/CoordinatesGrid/GridQualifier<br /></pre>I was part of the ABCD authors but at this time never looked much into the geospatial part of it. Now it looks to me that this is the case for a correct use of the "variable atomization" method that I did not like later on. Well, then there are cases when I like it. Of couse still work needs to be done on those concepts, and more important, people should start using them!</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div><br /></div>Javier de la Torrehttp://www.blogger.com/profile/03992185734194337802noreply@blogger.com6tag:blogger.com,1999:blog-5286956948868856311.post-18440037974258120342009-02-13T10:30:00.004+01:002009-02-13T10:41:26.045+01:001234567890<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj-Pj_5GuVA51pcEU2TS7jbILbcbFypYPBF7lqqmwfu-6p3fmxSdCX9_gr1oZsM4QDhdlm9Uv0ffb1wErMqyDT2qf0gTR6WdMjL83vT6nKJ0BMcVi4ccd2S5izItUk6ryBGHJpADDGxBc/s1600-h/1234567890.png"><img style="float:right; margin:0 0 10px 10px;cursor:pointer; cursor:hand;width: 400px; height: 193px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj-Pj_5GuVA51pcEU2TS7jbILbcbFypYPBF7lqqmwfu-6p3fmxSdCX9_gr1oZsM4QDhdlm9Uv0ffb1wErMqyDT2qf0gTR6WdMjL83vT6nKJ0BMcVi4ccd2S5izItUk6ryBGHJpADDGxBc/s400/1234567890.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5302214465541116530" /></a><br />Its friday the 13th and unix time will hit 1234567890 in a few hours. Well, in Continental Europe it will be saturday already, but anyone further west will be able to enjoy both at exactly Sat Feb 14 00:31:30 CET 2009Anonymoushttp://www.blogger.com/profile/02525336976753861766noreply@blogger.com2