Statistics and Appreciation
Monday, December 21. 2009
So I've been working on the statistics tracking "RefStats" (which apparently I'll have to rename; naming conflict with another project) and have given a much fonder appreciation for those people who work with creating statistic tracking services for a living (or a side project). It's ridiculous.
You record a hit, and a time, and perhaps some possible extra information depending on what your application is actually tracking. But just with the hit and a time, there's an exorbitant amount of ways to display information gathered from this data. How many hits were received on Thursdays? What were the average number of hits per weekend days in the month of March? Which day/hour of the month had the most hits? What percentage of hits were counted on Monday, Tuesday, Wednesday, and Friday - comparatively?
As the designer/developer of such a tool, you need to be aware of these types of questions. But that's not even half of the challenge. You also need to try to anticipate how your customers/clients will be using your application, and which of these reports will be the most important to them (if you're going to release reports in stages rather than do it all at once). You also need to try to reuse your code, so hopefully one querying method can be used to gather multiple reports, rather than a single method for each report (which although would be nice for modularity, a lot of these things are doing practically the exact same query with just a different WHERE clause, or different fields in the SELECT clause).
There's also the thought of interoperability of the data. What if you want an API for your application? What if you want the charting tool to be able to pull data from your primary querying method that serves all other aspects of your site so you don't have to develop two (or more) separate entities that are essentially getting the same information but rendering it differently.
There sure are some intelligent people and teams out there, such as Dan Grossman (http://www.dangrossman.info) of W3Counter fame, Laurent Destailleur of the original king of web stats: AWStats, and of course there's the team behind Google Analytics. These are examples of some of the big boys of analytics and by no means on the same miniscule level that I am even attempting to recreate; but the sheer fact of the amount of planning and ingenuity that goes in to such a project is simply mind boggling to me.
Anyway, read more to view some example images of the web reports that I have thus far - some development is still needed (note the disabled form fields, and I need to add a range option even though it already works), and the UI is not complete.
I originally decided to use the ezComponent framework's ezGraph library for creating these bar graphs and pie charts (sorry, no pictures here of the pie charts). After finally getting the graphs to work, I tested it in Internet Explorer. No version of IE supports SVG. Although ezGraph can also export as GIF, the whole point of me choosing the library was because everything could be styled, and look very slick (and could be printed in the browser, unlike Flash). It was approved for me to move forward and the Department Heads would simply be required to use Firefox to view the graphs, but I will eventually switch to a JS-only based graphing component; I have a few in mind but have yet to decide. Google Charts API doesn't allow me enough options for labelling (just in case it was suggested) - and the charts must be able to be printed (therefore no Flash solutions).
As a reminder, here's the desktop client that the staff use to create these "hits". It was built using Adobe AIR and simply uses a RESTful interface to the website's backend with a local (SQLite) datastore in case of server outage with the ability to automatically sync upon a re-established connection. The UI needs to be updated for 2 reasons: (1)I thought they'd never need more than 6 buttons, someone wanted 9...it doesn't fit in an 800x600 resolution display (yes, I know...tell me about it); and (2)there's a bug with the hover event in AIR when escaping the mouse from the targeted hover element to a transparent canvas - it doesn't know it's escaped from the target element and will hold the hover in place (and the tooltip malfunctions because of it)...oh and I need to add a close button.
There's also an issue with the tooltip JS but it should be fixed in future updates of the 3rd party script (qTip).
You record a hit, and a time, and perhaps some possible extra information depending on what your application is actually tracking. But just with the hit and a time, there's an exorbitant amount of ways to display information gathered from this data. How many hits were received on Thursdays? What were the average number of hits per weekend days in the month of March? Which day/hour of the month had the most hits? What percentage of hits were counted on Monday, Tuesday, Wednesday, and Friday - comparatively?
As the designer/developer of such a tool, you need to be aware of these types of questions. But that's not even half of the challenge. You also need to try to anticipate how your customers/clients will be using your application, and which of these reports will be the most important to them (if you're going to release reports in stages rather than do it all at once). You also need to try to reuse your code, so hopefully one querying method can be used to gather multiple reports, rather than a single method for each report (which although would be nice for modularity, a lot of these things are doing practically the exact same query with just a different WHERE clause, or different fields in the SELECT clause).
There's also the thought of interoperability of the data. What if you want an API for your application? What if you want the charting tool to be able to pull data from your primary querying method that serves all other aspects of your site so you don't have to develop two (or more) separate entities that are essentially getting the same information but rendering it differently.
There sure are some intelligent people and teams out there, such as Dan Grossman (http://www.dangrossman.info) of W3Counter fame, Laurent Destailleur of the original king of web stats: AWStats, and of course there's the team behind Google Analytics. These are examples of some of the big boys of analytics and by no means on the same miniscule level that I am even attempting to recreate; but the sheer fact of the amount of planning and ingenuity that goes in to such a project is simply mind boggling to me.
Anyway, read more to view some example images of the web reports that I have thus far - some development is still needed (note the disabled form fields, and I need to add a range option even though it already works), and the UI is not complete.
The default display, shows a sum of all calculated "hits" per month (for monthly reports by department heads or by director to the board).
Once a desk type is chosen from the dynamic dropdown (ala site profile for Google Analytics), the stats are then recalculated to show only that desk's stats, but also gives the option to view any submitted notes from the clerks that can then be read and deleted.
A breakdown of hits per question type.
The full out breakdown...the total questions broken down by question type, and then broken down by hour in a day, day in a week/month, or month in a year. The charting will automatically switch from a bar graph to a line graph if there are too many points to plot for it to be even remotely readable otherwise.
I originally decided to use the ezComponent framework's ezGraph library for creating these bar graphs and pie charts (sorry, no pictures here of the pie charts). After finally getting the graphs to work, I tested it in Internet Explorer. No version of IE supports SVG. Although ezGraph can also export as GIF, the whole point of me choosing the library was because everything could be styled, and look very slick (and could be printed in the browser, unlike Flash). It was approved for me to move forward and the Department Heads would simply be required to use Firefox to view the graphs, but I will eventually switch to a JS-only based graphing component; I have a few in mind but have yet to decide. Google Charts API doesn't allow me enough options for labelling (just in case it was suggested) - and the charts must be able to be printed (therefore no Flash solutions).
As a reminder, here's the desktop client that the staff use to create these "hits". It was built using Adobe AIR and simply uses a RESTful interface to the website's backend with a local (SQLite) datastore in case of server outage with the ability to automatically sync upon a re-established connection. The UI needs to be updated for 2 reasons: (1)I thought they'd never need more than 6 buttons, someone wanted 9...it doesn't fit in an 800x600 resolution display (yes, I know...tell me about it); and (2)there's a bug with the hover event in AIR when escaping the mouse from the targeted hover element to a transparent canvas - it doesn't know it's escaped from the target element and will hold the hover in place (and the tooltip malfunctions because of it)...oh and I need to add a close button.
There's also an issue with the tooltip JS but it should be fixed in future updates of the 3rd party script (qTip).
Desktop Client
Trackbacks
Trackback specific URI for this entry
No Trackbacks
Comments
Ideally you would know what your client(s) want from you. I was asked to create this project. When it finally came time that my schedule would allow the planning stage to begin, I then asked what type of stats they would want in a report and gave some possible examples. The response was, "Yeah, that sounds good. Do something like that." Oh man! :D
I'm a big fan of amCharts right now. I really like their look and how they handle making just about any data look good (like automatically figuring out how many labels to show when you define too many to actually fit into the graph area).
http://www.amcharts.com/
There's also a pure JavaScript graph library that's become a bit popular recently:
http://www.highcharts.com/
I'll have to take a look in to why some flash elements don't print for me on my site; it might be something as simple as my site's print style sheet since I had created it so long ago. Thanks for pointing that out.