The Flatmerge Blog has moved to
http://flatmerge.blogspot.com
Oh yeah, and check out our Kickstarter project. Back it and get some storage space and support.
Tuesday, April 15, 2014
Thursday, April 11, 2013
Exploring Hospital Patient Survey Data With Excel GeoFlow
As we add more OData features to FlatMerge, we are having fun discovering new ways to work with data. This post highlights 3 interesting things:
Then, to keep things simple , we add just one interesting data point, so select/click the PctRpt_YES_would_definitely_recommend_the_hospital column. That column appears in the Height dropdown. Select that Height again and choose "Average" to get the average respondent percentage who chose that that would definitely recommend the hospital.
We also clicked the "Chart" button to get a chart of the values. Drag around the map and have fun!
We hope this shows some interesting features of FlatMerge's OData support and how easy this functionality makes it to use with other tools like Excel and it's new toys like GeoFlow.
- The newly added FlatMerge OData $metadata support enables opening FlatMerge data directly into Excel Data Explorer. (note that FlatMerge offers minimal, but essential bits of support for $metadata currently)
- Some interesting data that is geocoded using Bing in Excel! Survey of Patients’ Hospital Experiences (HCAHPS) from data.gov.
- GeoFlow for Excel. From Microsoft: "GeoFlow lets you plot geographic and temporal data visually, analyze that data in 3D, and create interactive "tours" to share with others."
Here's how we loaded data from FlatMerge into Excel and used GeoFlow to see which hospitals are recommended by (sampled) patients, exploring using 3D maps!
First, we grabbed the text data from data.gov and changed the column names (they were way too long, like sentences). We also replaced "Not Available" to empty so calculations will be happy. Numbers were text percentages appended with %, so we removed that too.
Get the data and OData link here
Then we copied the OData link (http://flatmerge.com/Data/odata/9548f35f-7c02-4bca-a2e9-61d2b407fe92) and pasted it into the Excel Data Explorer -> From Other Sources -> From OData Feed
After clicking Apply, Excel queries the metadata about this OData feed, which just includes this one data feed (the survey data we've uploaded to FlatMerge). In FlatMerge, we provide one OData URL for each data file. So, choose the only table in the feed, which is identified by the GUID assigned by FlatMerge. Click that and Excel queries the data to pull the top 100 rows ($top=100).
Click Done to let DataExplorer get the rest of your data. Once DataExplorer finishes downloading the rest of your data, click Insert tab -> Map (Map is the GeoFlow launcher, download it and install it here http://office.microsoft.com/en-us/download-geoflow-for-excel-FX104036784.aspx).
Start invstigating the data, visually, in 3D! One of the coolest things about GeoFlow is that you can make time series interactive tours of your data. This hospital survey that we grabbed is just one survey (there may be more), so no time series tour here, but the 3D navigation is still pretty cool.
Select "State" column for the Geography, then click Map It. GeoFlow calls Bing and geocodes the States' geographic centroids.
Then, to keep things simple , we add just one interesting data point, so select/click the PctRpt_YES_would_definitely_recommend_the_hospital column. That column appears in the Height dropdown. Select that Height again and choose "Average" to get the average respondent percentage who chose that that would definitely recommend the hospital.
We also clicked the "Chart" button to get a chart of the values. Drag around the map and have fun!
Monday, March 18, 2013
New Feature: Get OData Output From FlatMerge
We've added OData output through FlatMerge. Now if you have a rectangular data file you have OData too! This is an addition to the current output formats of plain XML and JSON.
We'll be doing a series of posts about the many cool things you can do with this new feature. Today, we demonstrate the usefulness of OData and FlatMerge by loading some data into Excel 2013 directly from a FlatMerge URL.
Since earthquake data is so interesting, we've grabbed the freshest earthquake data from data.gov (https://explore.data.gov/Geography-and-Environment/Worldwide-M1-Earthquakes-Past-7-Days/7tag-iwnu), and uploaded it to FlatMerge http://flatmerge.com/Data/Details/fa05d620-96fc-4e18-8d7f-e9ba0ec59c75. (You can upload data for free to FlatMerge too!)
To get OData from FlatMerge, use the proper HTTP Accept header, like this
Accept: application/atom+xml
or add the querystring parameter &format=odata
Excel automatically does this (adds the Accept header) when you choose to add OData. So, to import FlatMerge OData into Excel:
1. Go to the FlatMerge details page of the data you've uploaded or are interested in and copy the OData URL as circled in red in the above image. Open Excel and navigate to Data->From Other Sources->From OData Feed (or PowerPivot->Manage->Get External Data->From Data Service->From OData Data Feed), and paste that URL into the Location Field.
We'll be doing a series of posts about the many cool things you can do with this new feature. Today, we demonstrate the usefulness of OData and FlatMerge by loading some data into Excel 2013 directly from a FlatMerge URL.
Since earthquake data is so interesting, we've grabbed the freshest earthquake data from data.gov (https://explore.data.gov/Geography-and-Environment/Worldwide-M1-Earthquakes-Past-7-Days/7tag-iwnu), and uploaded it to FlatMerge http://flatmerge.com/Data/Details/fa05d620-96fc-4e18-8d7f-e9ba0ec59c75. (You can upload data for free to FlatMerge too!)
To get OData from FlatMerge, use the proper HTTP Accept header, like this
Accept: application/atom+xml
or add the querystring parameter &format=odata
Excel automatically does this (adds the Accept header) when you choose to add OData. So, to import FlatMerge OData into Excel:
1. Go to the FlatMerge details page of the data you've uploaded or are interested in and copy the OData URL as circled in red in the above image. Open Excel and navigate to Data->From Other Sources->From OData Feed (or PowerPivot->Manage->Get External Data->From Data Service->From OData Data Feed), and paste that URL into the Location Field.
2. Select the one and only table listing. This is the original file name that was uploaded to FlatMerge.
3. Click Next and save the connection if you'd like. Select how to view the data (we've only tested with Table) and where to put it.
4. Data is imported! (Depending on the size of your data, this may take just a second, or hours).
Please try it out and let us know how you like it.
Monday, March 4, 2013
What's in my data?
One of the coolest things about FlatMerge is that it tells you types and lengths of your data. We're relying heavily on this feature as we build in dynamic OData output (coming soon!). So, browsing around data.gov, we found some interesting raw data about Congressional campaign expenditures https://explore.data.gov/Elections/2009-2010-Candidate-Summary-File/38zs-22s9
Loading this data into FlatMerge (http://flatmerge.com/Data/Details/77e6b770-8ef6-4658-912a-a771d159f270) shows some interesting things...
The description of the data on data.gov says that ZIP Code (can_zip) is 5 digits.
FlatMerge found some ZIP codes to be 9 digits... is this ZIP+4?
Also, it seems strange that the Candidate Incumbent Challenger Open Seat (can_inc_cha_ope_sea) field indicates that it is a 10 character field, but has an explanation for some letter abbreviations.
Turns out, the explanation of the abbreviated data is not needed, because the data contains the full text.
So, the documentation does not exactly match what's truly in the data.
Sunday, February 24, 2013
Use Excel 2013 with FlatMerge
In Excel, click Data tab -> From Other Sources -> From XML Data Import
Select the Data query location from FlatMerge, here we open the most recent Earthquakes and choose format=xml
Excel 2013 is ready for import
Have some Excel 2013 fun!
Friday, January 11, 2013
Farmer's Market Map
We've created a Google Map of Farmer's Markets using data in FlatMerge. This post shows how FlatMerge makes it simple.
1. Grab the csv data from data.gov and upload it to FlatMerge.
2. Browse Google Maps API examples for some cool ways to make maps.
3. Create a lean OData query to get the fields I need.
http://flatmerge.com/Data/Query/31974ef9-5a62-420d-a6c3-945b3ab8e6d4?$select=x,y,website
1. Grab the csv data from data.gov and upload it to FlatMerge.
2. See what's in the data. X, Y (longitude and latitude coordinates, cool!), and website are good for now.
2. Browse Google Maps API examples for some cool ways to make maps.
3. Create a lean OData query to get the fields I need.
http://flatmerge.com/Data/Query/31974ef9-5a62-420d-a6c3-945b3ab8e6d4?$select=x,y,website
Load markers on the map using the x/y coordinates from the JSON returned by FlatMerge query:
| $.getJSON("http://flatmerge.com/Data/Query/31974ef9-5a62-420d-a6c3-945b3ab8e6d4?$select=x,y,website", function (data) { | |
| for (var i = 0; i < data.length; i++) { | |
| var market = data[i]; | |
| var myLatLng = new google.maps.LatLng(market.y, market.x); |
4. Check it out http://flatmerge.com/markets.html! View the source code to see how simple it is. (Note that the code won't work using a local .html file, it must be on a real web server or localhost.)
Mobile app and how-to post coming soon...
Sunday, December 9, 2012
Earthquake list for the past 7 days
(Quick Note: direct link to this data:
http://flatmerge.com/Data/Details/8ef2ad73-660a-4228-bb8f-eedeae821c1c,
or another interesting set of data about Farmers Markets:
http://flatmerge.com/Data/Details/31974ef9-5a62-420d-a6c3-945b3ab8e6d4)
For the first FlatMerge post, we've shared some data with you about earthquakes. From data.gov:
https://explore.data.gov/Geography-and-Environment/Worldwide-M1-Earthquakes-Past-7-Days/7tag-iwnu
Since this is the first data loaded into FlatMerge by anyone, this post is also a how-to for getting started with FlatMerge.
Uploading your data is simple. The file source (currently supported) is a comma-delimited text file with column names in the first row.
FlatMerge ingests the file then sends you to your data list page. Your new data's status will say "Analyzing" while FlatMerge detects the exact data types and width's of each column.
Refresh the data index page until your data status is "Complete". The number of rows and columns are also listed on this page. To get into your data, click the Details link.
http://flatmerge.com/Data/Details/8ef2ad73-660a-4228-bb8f-eedeae821c1c
Here you will see your data details in FlatMerge. Listed are: date/time created (GMT), the original file name, number of rows, and storage space used. Since FlatMerge optimizes your data, the size (Kb) will be much less than the actual file size uploaded.
http://flatmerge.com/Data/Details/8ef2ad73-660a-4228-bb8f-eedeae821c1c,
or another interesting set of data about Farmers Markets:
http://flatmerge.com/Data/Details/31974ef9-5a62-420d-a6c3-945b3ab8e6d4)
For the first FlatMerge post, we've shared some data with you about earthquakes. From data.gov:
https://explore.data.gov/Geography-and-Environment/Worldwide-M1-Earthquakes-Past-7-Days/7tag-iwnu
Since this is the first data loaded into FlatMerge by anyone, this post is also a how-to for getting started with FlatMerge.
Upload
http://flatmerge.com/Data/UploadUploading your data is simple. The file source (currently supported) is a comma-delimited text file with column names in the first row.
FlatMerge ingests the file then sends you to your data list page. Your new data's status will say "Analyzing" while FlatMerge detects the exact data types and width's of each column.
http://flatmerge.com/Data/Details/8ef2ad73-660a-4228-bb8f-eedeae821c1c
Here you will see your data details in FlatMerge. Listed are: date/time created (GMT), the original file name, number of rows, and storage space used. Since FlatMerge optimizes your data, the size (Kb) will be much less than the actual file size uploaded.
Three links will also be listed:
Location: this is the location of your data details
Data query location: this is the location of the actual data. This URL can be used from other clients for use in applications, web pages, or any way you wish to share or use your data.
Metadata location: this is the URL of the metadata that FlatMerge created. After uploading, FlatMerge analyzes your data and discovers the true data types and widths (for number and character columns). If numeric, the whole number and decimal parts are discovered. The metadata can also be queried using OData URLs.
Data query location: this is the location of the actual data. This URL can be used from other clients for use in applications, web pages, or any way you wish to share or use your data.
The data can be queried using OData URL conventions and returned in JSON or XML. (Currently, the supported queries are: top, orderBy, Select, Skip).
Metadata location: this is the URL of the metadata that FlatMerge created. After uploading, FlatMerge analyzes your data and discovers the true data types and widths (for number and character columns). If numeric, the whole number and decimal parts are discovered. The metadata can also be queried using OData URLs.
Also on the data details page is a grid that displays the first 1,000 rows of data. For this grid, we used the open source Kendo UI Grid.
And below the data grid is a table of metadata discovered.
Share
Most importantly, share your data! We want to make it easy to share data in a way that assures others of data quality and transparency.
The social media sharing links point directly to your data's details.
Subscribe to:
Posts (Atom)