I was randomly perusing the web and stumbled across an older interesting List Apart article (September 19, 2005 to be exact). In it, the author used JavaScript to create a list of all links found on the page, and then create a footer with a footnote-like list. I was using simple CSS to display the URL of the hyperlink in print display media, but that doesn't work in some browsers (IE), and worse off it can really mess up the visual order of the page.

Inspired by the article's intentions, I decided to use some jQuery to whip up a similar solution. Differences? I exclude hyperlinks set to the hash symbol (#) which shouldn't be found anyway (graceful degredation people!), mailto links, and I don't find link element's citations (blockquote "cite" property). I also believe his script would grab image information, but I'm not entirely sure, mine only searches hyperlinks. I also opted not to exclude listings of duplicate hyperlink values. If it's found twice, it lists it in the footer twice. Less JS processing, and more clear to the visitor (in my opinion).

Expected Issues: It requires JavaScript. If JavaScript is enabled but CSS is disabled, the hidden content will be visible to the user.

On to the code!

First thing's first, we should find the links in our code that we'd like to target. For my situation, I have a content class defined on my page for all content, and the column that contains the actual body is called ".col2_right" (yes, I know, ignore the fact of the poor naming scheme). So, in jQuery, we're going to target all hyperlinks found in the col2_right and content classes, but we don't want to include links to hashes or mailto links. I've also removed listing to nofollow links, but that is my choice (probably not a good one, but I digress).

var links = $('.col2_right.content a:not([href^=#],[href^=mailto],[rel=nofollow])');
We'll also need some sort of container to keep these found links wrapped in. I chose to use a fieldset and legend, you could just as easily create a DIV and a H2 or other tags for your own purposes. I also want to create an ordered list to match the found hyperlinks.
var footnotesWrapper = $('<fieldset>', { css: { clear: 'both' } }).addClass('print_only'); var footnotesLabel = $('<legend>', { text: 'Website Addresses Used in the Document' }).appendTo(footnotesWrapper); //create an OL to hold the footnotes var footnoteList = $('<ol>').appendTo(footnotesWrapper);
Here comes the real center, meaty goodness of the script: looping through our list of found hyperlinks, creating a little notification text next to the hyperlink's text, and add the hyperlink URL itself to the footnote section in the ordered list as a list item.

We use jQuery's each() method to loop through the items in the array, grab the link's URL so that we can do some testing on it, and modify it's value for inclusion in to the footnotes. Some links may be relative URLs, but that doesn't mean much to someone who doesn't know what a relative URL is. In fact, it would probably confuse me too at first. I happen to know that my domain forces www for the subdomain so I've added that in, fix that up as you like. I then create a span tag to hold the associated footnote number (the ListApart article used a superscript tag, I found regular text is more easily readable), add a class to it so it's only visible when printed, and create my list item element with the URL from our link, and place those elements in to the document in the appropriate places with the appendTo jQuery function.
$.each(links, function(i){ var linkText = $(this).text(); var linkValue = $(this).attr('href'); if(linkValue.substring(0,1) === '/'){ linkValue = 'http://www.'+document.location.host + linkValue; } //create element to hold span with class to hide except on print var newElement = $('<span>', { text: ' ['+ ++i +']' }).addClass('print_only').appendTo($(this)); var listEntry = $('<li>', { text: linkValue }).appendTo(footnoteList); });
Last but not least, we append our entire footnote to the end of our content body section. The full code is below.
$(document).ready(function(){ //get the container and target var links = $('.col2_right.content a:not([href^=#],[href^=mailto],[rel=nofollow])'); //create a container and heading for the footnotes var footnotesWrapper = $('<fieldset>', { css: { clear: 'both' } }).addClass('print_only'); var footnotesLabel = $('<legend>', { text: 'Website Addresses Used in the Document' }).appendTo(footnotesWrapper); //create an OL to hold the footnotes var footnoteList = $('<ol>').appendTo(footnotesWrapper); $.each(links, function(i){ var linkText = $(this).text(); var linkValue = $(this).attr('href'); if(linkValue.substring(0,1) === '/'){ linkValue = 'http://www.'+document.location.host + linkValue; } //create element to hold span with class to hide except on print var newElement = $('<span>', { text: ' ['+ ++i +']' }).addClass('print_only').appendTo($(this)); var listEntry = $('<li>', { text: linkValue }).appendTo(footnoteList); }); // append the heading and <ol> to the target $('.col2_right.content').append(footnotesWrapper); });

Oh, and the CSS: ".print_only { display:none; }" This should be placed in your normal style sheet, and leave it out from your print style sheet. If you don't have a print style sheet, extend it a bit: @media print { .print_only { display:none; } }

In CakePHP v1.3.x, there will no longer be an automatic dump of tons of information (probably because the DebugKit plugin can handle all of that information for us) when in debug mode. However, in my non-CakePHP websites, and even in my Cake websites, I have become accustomed to viewing the source to see a full render time for the page in an HTML comment. Thanks to the help of "jnay" over at CakeQs.org, I've found out about a new (as of PHP 5.1) $_SERVER variable argument called "REQUEST_TIME". It holds the UNIX timestamp of the initiation of the call.

By using this variable, along with the optional boolean argument for microtime (as of PHP 5.0.0), we can determine the HTML render time of our application with the following calculation in either our footer view/template, or controller (preferably in the last possible iteration of your project's render):
<?php echo microtime(true) - $_SERVER['REQUEST_TIME']; ?>
The echo is optional and used only as an example here; it simply depends on how you use the calculation in your script(s)/framework(s). This is much easier than starting the timer in the header and calculating the difference in the footer like I had to do in the PHP 4 days!

Remember, this is only for trivial estimations and comparisons. If you need to thoroughly evaluate your script(s) for time efficiency you should be using something else, such as Xdebug and a form of CacheGrind for your choice of OS.

I decided that it was finally time I took a serious look at increasing the YSlow score for our websites. I was under the impression that browsers would automatically cache (by default anyway) certain graphic elements to save on bandwidth, but our staff website has a repeating background image for a left hand navigation bar, and it would disappear for a fraction of a second while reloading the other page; this told me that something was not right, and YSlow might be the answer I was looking for (and that my PHP framework of choice was not to blame).

YSlow:
Initial Score: 67

I choose the smallest size images exported from Photoshop, and if I ever use a PNG, I use a great tool called "PNGGuantlet" (Windows only) which compresses PNGs while also removing the alpha channel problems that make PNGs appear with different colors than they were intended to. I also have compressed my main JS file, and CSS file(s). All XHTML should be valid (unless I updated a link and forget to escape an ampersand), the DOCTYPE is listed, character set is given, and all non-background images contain height and width properties. Oh, I also have a default favicon.ico file. These are all good recommendations to do regardless, and in doing so YSlow gave me a pretty decent score (albeit, not that great if it was a test in school).

Two modifications to YSlow's scoring:
1. I am not Amazon. I don't necessarily need a CDN for my content.
2. I don't need a subdomain to house my CSS, JS, and images. If I end up storing videos, that already has a subdomain set up and ready.

Therefore, I modified YSlow's grading scheme. To do so, there is a dropdown menu to the right of the 4 tabs in the YSlow toolbar, it is labelled "YSlow(V2)". Click the "Edit" button to the immediate right of that. In my scheme, I disabled "Use a Content Delivery Network (CDN)" and "Use cookie-free domains". You cannot overwrite the rule set, so you have to save it as something else, I named mine "SSPL" since that is the website domain that I care about (for work). Different websites have different criteria, so it only makes sense to name it something related to the website you are concerned about. Go through and uncheck what you don't believe applies to you. If you're uncertain, leave it checked. This little configuration raises my grade up already - that was easy.

Here are the .htaccess rules I've added to raise my own score, what they're for and why I chose to make them:
# Optimization settings AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript text/css application/javascript text/json application/json Header unset ETag FileETag None ExpiresActive On ExpiresDefault "modification plus 10 years"

Enable GZip'ing of components:
The AddOutputFilterByType DEFLATE [list of mimetypes] was the command I used to gzip the particularly listed files that have such a MIME-Type: html, txt, js, css, js (variation), json, json (variation). Some browsers read JSON and JavaScript as an application mime-type whereas others read it as a text type, I figured I would simply cover my bases. Although these are ASCII types, you could just as easily gzip binary types, such as GIF, JPG, or PDF (though theoretically they should already be compressed and you're just adding extra work for the server).

Configure ETags:
From what I have read, ETags are a mixed breed. They can have some benefits, but they can also have some detractions. Again, considering that I'm not Amazon and I will be enabling the Expires heading, I opted to simply disable ETags. You can disable ETags in Apache under an .htaccess file with the unset rule above, along with the FileETag None to cover most supporting browsers (one would hope). I wasn't able to find any examples on how to properly "configure" ETags so that YSlow would not complain other than to simply turn them off. ETags, from what I've read, are a variation on telling the browser to cache certain files in a certain manner, but it doesn't work on any website that is run on multiple servers for the same content. Either way, I opted for the Expires heading which made ETags rather redundant from what I could tell.

Enabling an Expires Heading:
It appears that in Apache, by default, Expires headings are disabled, and just like mod_rewrite, you must explicitly tell Apache that you wish to use it, so the first thing you have to do is turn it on: "ExpiresActive On". Once that is done, there are a few ways to set the expiry time; I opted to use a human readable implementation, and YSlow expects a "far future" date (from the date last modified). As one of my sites is currently built with HTML files pulling in a template, the original files won't typically be modified for a long time, so I had to make my future date a FAR future date (10 years from last modification) - your experience may vary. For further information on Expires, take a look at the Apache documentation for mod_expires.

Additional speed enhancements:
I recently read that if you have a framework that may take some time figuring out what data must be displayed prior to even getting it to the server, it may be a good idea to flush the buffer so that the data that can be sent to the browser can be sent prior to everything else - such as the document head which contains the page title, CSS, and sometimes JS. That way, while the rest of the page is being rendered by the server, the JS/CSS has time to download. This is actually why I decided to leave my JS at the top of my website rather than placing it in the footer. I live by graceful degredation, but if the page loads before the JS does, it can also cause a screen flicker which is against accessibility guidelines for HTML. In this case, it was possible speed for accessibility - I choose accessibility. I personally really don't want me, or my client(s) getting sued for accessibility issues because "Oh hey, it loads a second faster now!". If you had an interactive rich website such as Facebook or MySpace, I'd imagine you'd have to place JS in the footer and have a separate, accessible domain for others...if that's even easily possible. Anyway, what I did with flushing the output buffer was to (in PHP) modify my template file, I called PHP's flush() function after the closing HEAD tag in the HTML template file.

Anyway, with those small tweaks and tricks, I was able to get my score up to 96. I have a B in 3 categories:
1. Expires header issue (CDN of ajax.googleapis.com, it won't recognize my CDN) as a CDN has a non-far future expiration date in the called JS file.
2. Minify CSS and JS - it is minified, so I'm not sure what it wants from me. The filesize is probably too large for an A.
3. Put JavaScript at the Bottom - I already explained why I don't want to do this, but I've left the rule in anyway.

Now my site seems to be a bit more responsive, and the background image I mentioned before no longer disappears and reappears (unless I press refresh really, really fast, confusing the browser's cache I guess).

Cassie is a PC Reservation Management system software built for libraries, or other venues that wish to offer multiple PCs for the public to use. It also has a print management station. Unfortunately, as of the current version, when a user goes to view their own print jobs, their account window stays active "forever" until they, or someone else clicks on the "Close" button. When using with a JAMEX corporation coinbox, it also doesn't give the user their change until the window is closed (insert $1.00 for a $0.60 charge, it won't expell the $0.40 until the window is closed).

This is a bit of a problem.

I've used AutoIT, an automation scripting language, to automate the process of a user clicking on the close button. The makers of Cassie didn't make it easy though, I was unable to detect the actual button, thankfully ALT+F4 worked (ESC worked until the JAMEX portion was added, for some reason it stopped working afterward).

#include <Timers.au3> ;declare variables $idletime = 0; $seconds_to_wait = 45; $action_time = 1000 * $seconds_to_wait; while 1 $idletime = _Timer_GetIdleTime(); If $idletime >= $action_time Then WinWait("[CLASS:#32770; TITLE:Show Print Jobs]"); If Not WinActive("[CLASS:#32770; TITLE:Show Print Jobs]") Then WinActivate("[CLASS:32770; TITLE:Show Print Jobs]"); WinWaitActive("[CLASS:#32770; TITLE:Show Print Jobs]"); Send("!{F4}"); issues an ALT+F4 command via mimicked keyboard input EndIf Sleep(100); wait 100 milliseconds before checking again WEnd

This uses the included Timers.au3 file to do most of the dirty work in detecting an idle state. If the computer is left idle for 45 seconds, it will issue ALT+F4 and close the window. ALT+F4 cannot close the application itself in this instance (and won't be issued unless that window exists and is active anyway). Although it checks every 100 milliseconds and runs at all times that the computer is turned on, I tested the resources it takes up...it was barely noticeable on an old P4 @ 2.80 GHz w/2.86 GB RAM (my work PC). This application (once compiled with AutoIT) should be placed in the Windows' Startup folder (or a respective registry key).

I had a momentary stumble today that I (somehow) couldn't get through my head. Perhaps it's just because it's been so long since I've worked with mod_rewrite, but regardless of that, here's the situation and the solution:

"The Situation" said that the situation is:
I had multiple conditions that needed to be met before a rule should be run, otherwise it should be ignored.
I am running (and require) SSL on my production server.
I am not running SSL on my development box.
I didn't want multiple .htaccess files in my SVN repository.

The solution:
"Duh." Multiple RewriteCond in a row (separated by a newline) all must be met before the RewriteRule is run.

The resulting .htaccess took the following form:
<IfModule mod_rewrite.c> RewriteEngine On RewriteCond %{HTTPS} !=on RewriteCond %{HTTP_HOST} ^employees\.sspl\.org [NC] RewriteRule ^(.*) https://%{SERVER_NAME}/$1 [R,L] RewriteCond %{REQUEST_FILENAME} !-d RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^(.*)$ index.php?url=$1 [QSA,L] </IfModule>
The last 3 lines dealing with rewrite in my .htaccess file are related to the code library I'm using, and are not pertinent to the problem here. The important lines are as follows:
RewriteCond %{HTTPS} !=on
RewriteCond %{HTTP_HOST} ^staff\.sspl\.org [NC]
RewriteRule ^(.*) https://%{SERVER_NAME}/$1 [R,L]

The first condition tests to see if SSL (https protocol) is not being used in the request.
The second condition tests that the domain's host is "employees.sspl.org" (URL changed to protect the innocent).
If both tests pass, then I redirect the user to the host (and path "/$1") while using the https protocol (SSL). Done.

Twitter can be a real boon for learning new things and finding interesting little projects so long as you manage to follow the right people.

Sticky Footer: Alternative Approach - This takes an alternative approach using only a single conditional statement for IE and supports IE5.5+, with 100% valid CSS and HTML

Design Contract Template - A template for contractual jobs in design, alter to your own use (quite verbose)

15 Tips for Designing Tables - A nice resource on using CSS and JS to markup and add interaction to your tables, a nice new take on it with good examples

A review of Rich Text Editors - Not just your grandma's WYSIWYG Editor...great thorough review

An HTML5 Video Player - Created with JavaScript and HTML5 markup; no Flash here folks!

Microsoft's Translator Widget - This thing is the coolest translator I think I've ever seen. It loads the translated version progressively via AJAX without any new page load -- with a loading bar to boot!

Free SEO Toolkit for IIS - I don't use IIS, but for those that do, this would be an enormous help, watch the video to see (good to watch even if you don't use IIS)

Dygraphs - Interactive, zoomable, plottable, QUICK JavaScript-based graphing library - it might give Highcharts.js a run for its money on line graphs (it's basically all this does, unfortunately)

Vector Social Media Icons - I'm not usually one to be excited enough about icons to tell people about them, but the fact that the author supplied two versions of the icons in vector format is pretty darn cool (32x32 and 16x16; 32x32 never scales well to 16x16 size, and 16x16 size is made especially for that size and doesn't look good stretched out)

Revised Font Stack - Looking for more fonts to use in your CSS without requiring @font-face or sIFR? OS's have updated their default font base, check here to see what more options us web developers might have nowadays

jQTouch - "A jQuery plugin for mobile web development on the iPhone, iPod Touch, and other forward-thinking devices." Check out the demo, it's sick...in the good sense.

Free App A Day.com - For all you iPod Touch and iPhone users, check out this site for some free apps! (Don't say I never gave you anything!)

So I worked on the CakePHP plugin for Prowl a bit more, and in doing so, I created shell-based access to the plugin. I can now send Prowl messages from the console (i.e.: I can run PHP via cron to check on things and send myself the results, if necessary). A shell task I created was for server load, as a followup to my previous post. In working this out, I've modified the output for CakePHP's Shell::out() method so that it auto-wraps and indents a bit more nicely (my ticket in the LighthouseApp is here). Time for some output examples:

Issuing the Load Alert help call: cake prowl load_alert help
--------------------------------------------------------------- CakePHP Prowl Load Alert Console Task Usage: cake prowl load_alert <params values> <args> Description: This task tests the current server load and issues a Prowl message (if required) to the passed parameter list of Prowl APIs indicating the server load. --------------------------------------------------------------- Parameters: -silent <false> Commands: help Shows this help message. Arguments: apikey Up to 5 comma separated Prowl API keys

Sending a message is as simple as: cake prowl msg [apikey] ... it can get as advanced as cake prowl msg [apikey] [priority] [application] [event] [description]

The output for running the load alert command?
Checking for high server load... Server load: 4.81 5.25 6.44 Server is running with an acceptable load.
If it's above my arbitrarily set values, it'll send a different warning depending on the level to my Prowl-enabled device. By sending a "-silent 1" parameter to the command message, output is disabled (perfect for cron).

Please note that I don't have any styling for shell output (something I'll have to fix) so the pasted character spacing and newlines won't work as expected in my example output.

I've signed up for "Project 52", and I've already missed last week, only the third week in. That being said, the whole point of Project 52 is to get people using their blog, to post more. There is a goal, but the ultimate goal is simply to post more.

A lot of my lack of success is simply trying to come up with real valuable content each and every week. I must realize that it's simply not going to be possible - at least not up to my own standards. That's not to say that I can't still post interesting things, just not necessarily the type of stuff I'd want to write about with gusto and enthusiasm - but still kinda cool. I mean, we learn things on an almost daily (if not hourly) rate as web developers. Why can't I simply zero in on one and discuss it? Or, perhaps mention all the neat things I've found in a particular week if I simply don't have time to write a full article on something I'm working on (which is what happened this past week - I was shipping furniture to my new house in a rental truck from out of state).

Anyhow, here are some interesting links that I've found (mostly from Smashing Magazine's Twitter account) within the past week or so (in no particular order):

- Why Mood Boards Matter: A look in to a quick way of getting a client on board with your end-result design without actually showcasing any direct portion of the site. An interesting approach that I had heard of and seen, but this iteration looks much nicer and more professional (the 2nd example).

- PostageApp: "The easier way to send email from web apps". This service offers a (currently) freely available (beta) API to send emails, also allowing for reporting services and debugging for such tasks. Quite the interesting little project, and could be extremely useful and handy.

- Uptime Robot: I currently am using mon.itor.us for my website uptime reporting. However, I originally signed up for the IM chat based alerting, and it only worked for the first two-three weeks. They adjusted the alerting mechanism to only alert via email when it had X number of consecutive bad pings...it might be time to evaluate a competitor, and here's one of them!

- jQuery Sheet: To quote their own description - "jQuery.sheet addresses the web-based spreadsheet or "websheet" in a different, simple, approach - using html, you simply load the sheet from a url, and you save it to a url... Simple. Or write your own approch. jQuery.sheet manages the sheet creation, viewing, and editing processes, you simply build your application around it." Quite interesting, and although I don't currently have a use for it, it's definitely something that I could see being very useful and powerful.

- Project Sikuli: This project is quite different. It appears (from a quick glance) to be a scripting language to help automate tasks. That's nothing new. The new part is the fact that it uses screenshot captures to do the scripting. Yes, that's correct. I honestly can't do it justice to talk about it, so check out the site and watch the video and read up on it if it sounds at all interesting.

- WhiteSmoke Writer 2010: The new version adds "writing review", which according to the quick animated Flash file on their homepage, might even be useful for me. It suggests changes to grammar to make a "negative sentence" neutral, or positive. This is something I sometimes have issues with when it was not intended. If I start a business of my own on the side, it's the last thing I want to do to my clients, so this might be worth a purchase, or at least an evaluation/consideration.

- SupportDetails.com: Although it's such an easy concept and execution (practically anyone could create this service), it's got a nice UI and...hey, I didn't think of it first. SupportDetails.com grabs the details of your current session data and allows you to mail it to someone. Let's say a website visitor is claiming there's this weird bug on your site, but you just can't see it. This could possibly identify the odd combination of software/resolution/settings that could cause the issue. At the very least, it's a simple and neat idea.

...and saved for last just because... The Ultimate PC Chair: I want one for work. Quoted from the linked blog discussing it: "NOTE TO MY BOSS – Dear Mr Boss, I am certain that my productivity would increase by 60,000% if you bought me this chair
DISCLAIMER: There is also a chance that I will be so comfy that I will fall asleep during work hours."

I have a shared Dreamhost account. In fact, I manage two: my own, and work's. Typically I don't too much care about my personal site going up or down, but I have multiple sites running under my account - and our work account with Dreamhost must maintain a decent responsiveness during business hours.

I've been developing a CakePHP plugin for Prowl but haven't quite polished it well enough for release. In the meantime, my work's Dreamhost account has had major issues where the server load would spike up to 350+ (I have no idea how many cores there are). That's just ludicrous. We've recently been moved to a new server by the support staff (for the 5th time) and although the average server load is now 0.40, I don't want to take my chances without a backup plan. Enter Prowl and my iPod Touch.

(To use Prowl, you would need an iPod touch or iPhone, and paid for and installed the Prowl application; it's $4.99 as of this writing. You then need to get your API from the prowl.preks.net website.)

We'll be using the PHP 3rd party API for Prowl, built by "Fenric", and we will go to his GitHub account to get it.

Once we have the PHP API for Prowl from Fenric, we'll build some quick code to make use of it:
<?php include('ProwlPHP.php'); //Fenric's Prowl API $load = sys_getloadavg(); //oh yeah, this'll only work on a *nix machine $current_load_avg = $load[0]; $priority = 0; //available options are -2, -1, 0, 1, or 2 $msg = ''; if($current_load_avg > 300){ $msg = 'SEVERE SERVER ALERT, LOAD ABOVE 300!!'; }else if($current_load_avg > 125){ $msg = 'AMBER ALERT: Server Load above 125.'; }else if($current_load_avg > 50){ $msg = 'Notice: Server load above 50.'; } if(!empty($msg)){ $prowl = new Prowl('your API key'); $prowl->push(array( 'application' => 'Server Load Alert', 'event' => 'Load Warning', 'description' => $msg, 'priority' => $priority ), true); } ?>
This little script can easily be setup with a cronjob and automatically check the current server's load and report issues. It won't report if a server is down since it must run on the server it is reporting on, but it would help inform about possible problems that should be looked at. This could easily be extended to check the MySQL long query logs, or other reporting functions.

So, I hear you like the idea of push notifications on the iPhone. Oh, what's that? You wish you could send push notifications from your website and/or iPhone enabled web app? Hey, didn't you know that you can?

Thanks to Mike, and his post about what John Blyberg's been doing, I got interested in Prowl. Prowl is an iPhone app ($4.99 from the iTunes store, I believe) that lets us send push notifications to iPhone or iPod Touch devices (and hey, it works with 1st gen devices too!). You would need version 3.1 of the iPhone OS, and jailbroken phones are not officially supported (some work, some don't due to how they were jailbroken), but for the most part "it just works", and it's fast. From my tests, the longest delay I've had thus far on a "normal" importance level Prowl message was about 2-3 seconds.

So what use does it have? I'm sure there could be quite a lot. There are forums for Prowl discussions, and one of the stickied topics is just about what people are doing with custom notifications, and there are a varied slew of responses. It's quite interesting, actually.

It's not much of a surprise to anyone that's been keeping tabs on this blog (hi Mike) that I have been getting acquainted with the CakePHP framework. Because of all the uses of Prowl, and my newfound love for CakePHP, I've decided to write a plugin specifically for Prowl in CakePHP. I am not the first to create extensions for Prowl by any means, nor am I even the first to think of getting it CakePHP-ready. I've found two others after having started my work that have created Prowl Components: Jacob Oehler Morrison, and Eric Holmes (note: as of this writing, Eric Holmes' domain seems to be expired). Hopefully I can add a little bit more zest and out-of-the-box capabilities to a plugin. If anyone has some ideas, let me know about it in the comments!

So I've been working on the statistics tracking "RefStats" (which apparently I'll have to rename; naming conflict with another project) and have given a much fonder appreciation for those people who work with creating statistic tracking services for a living (or a side project). It's ridiculous.

You record a hit, and a time, and perhaps some possible extra information depending on what your application is actually tracking. But just with the hit and a time, there's an exorbitant amount of ways to display information gathered from this data. How many hits were received on Thursdays? What were the average number of hits per weekend days in the month of March? Which day/hour of the month had the most hits? What percentage of hits were counted on Monday, Tuesday, Wednesday, and Friday - comparatively?

As the designer/developer of such a tool, you need to be aware of these types of questions. But that's not even half of the challenge. You also need to try to anticipate how your customers/clients will be using your application, and which of these reports will be the most important to them (if you're going to release reports in stages rather than do it all at once). You also need to try to reuse your code, so hopefully one querying method can be used to gather multiple reports, rather than a single method for each report (which although would be nice for modularity, a lot of these things are doing practically the exact same query with just a different WHERE clause, or different fields in the SELECT clause).

There's also the thought of interoperability of the data. What if you want an API for your application? What if you want the charting tool to be able to pull data from your primary querying method that serves all other aspects of your site so you don't have to develop two (or more) separate entities that are essentially getting the same information but rendering it differently.

There sure are some intelligent people and teams out there, such as Dan Grossman (http://www.dangrossman.info) of W3Counter fame, Laurent Destailleu​r of the original king of web stats: AWStats, and of course there's the team behind Google Analytics. These are examples of some of the big boys of analytics and by no means on the same miniscule level that I am even attempting to recreate; but the sheer fact of the amount of planning and ingenuity that goes in to such a project is simply mind boggling to me.

Anyway, read more to view some example images of the web reports that I have thus far - some development is still needed (note the disabled form fields, and I need to add a range option even though it already works), and the UI is not complete.
Continue reading "Statistics and Appreciation"

I was working on my Reference Statistics Tracker for work, and ran in to a snag.

I am using eZ Components' ezGraph class to create image-based graphs on the fly. It's fast, efficient, and it looks nice - it can even render in SVG (I'll be using JPG or PNG...thanks IE). For my particular needs, I wanted the exported dataset to have a specific formatting to the keys, to represent the different values in my graphs. For instance, let's say I have the following stats from 100 users in our company, and the Operating System they use. This is purely fictitious:

array('Windows' => 90, 'OSX' => 6, 'Other' => 4);

That specific array is crafted perfectly in the format that works well with eZ Components' Graphing class, for a simple bar chart, or a pie chart.

So now I just had to pull the values from the database in the format I wanted. Well my keys in this case were dates, or times (span of days, or hours in a day, or month in a year, etc...). The values were still numeric values in this case. So, we're looking at something like:

array('HH:MM:SS' => 10);

However, no matter how hard I tried, if there were no values inserted in to the database for a specific time, I could not return a value of null, or zero - even with an outer join. My next thought was to use something like array_merge() on the resulting dataset to fix it.

The PHP function of range() just didn't cut it. Although I could have converted the datetime values to timestamp values (either via SQL or PHP), and used the 3rd parameter of the function to add the specific number of seconds needed, I was weary of doing that as I remember reading about issues that could cause (adding seconds to a timestamp value calculate future time). Besides, I'd still have to use array_walk() or array_map() to format my array's keys the way I wanted; that left yet another function, and a line of code for the call to array_walk/map. (My host does not yet have PHP 5.3.x, which will allow for lambda functions in array_map.) So, I decided to create my own little function, and figured I'd post it here in case (1)someone could use it and it'd help them out, or (2)someone thinks there's a more efficient way of doing this and wants to give feedback.


<?php //$datetime_start - expects a strtotime compatible value //$datetime_end - expects a strtotime compatible value //$interval - can take either a strtotime compatible modifier (i.e.: -1 day), or an integer, // which will be perceived as total seconds and converted to a strtotime compatible modifier //$format - the format of the key for the array's key/value pair (see php.net/date for formatting options) //$value - the default value to be set for the blank array which will be used in a followup array_merge call. function datetime_range($datetime_start, $datetime_stop, $interval, $format, $value = 0){ if(is_int($interval)){ $interval = $interval . ' seconds'; } $range = array(); $datetime_start = strtotime($datetime_start); $datetime_stop = strtotime($datetime_stop); for($curTime = $datetime_start; $curTime <= $datetime_stop; $curTime = strtotime(date('r', $curTime) . ' + '.$interval)){ $range[date($format, $curTime)] = $value; } return $range; } //TIME FOR AN EXAMPLE OF USAGE: $myArray = array('10/01' =>3, '10/04' => 7, '10/05' => 4); $default = datetime_range('2009-10-01 00:00:00', '2009-10-05 00:00:00', '1 day', 'm/d'); $myArray = array_merge($default, $myArray); print_r($myArray); /************ this would have printed ************ Array( [10/01] => 3 [10/02] => 0 [10/03] => 0 [10/04] => 7 [10/05] => 4 ) ********************************************/ ?>
This could be useful for similar graphing/charting code, I don't think it's specific to eZ Components'. Perhaps it could be useful for something else that I can't think of. Have fun with it.

The Adobe AIR portion of the Reference Statistics Tracker is "done" (and has been for a week or two now). I just got approval over the extremely basic wireframe layout of the backend (web-based) tool that actually shows the stats data. Although it's basically only storing a very small amount of data, there are so many different ways that the data itself could be shown. If you think about how Web Analytics software can show so many different things; also think about what stats you have with it...it's basically a huge (textual) database of single line entries - the address, the referrer, the time, the visitor's IP, and a response code...yet there are tons of different ways to show that data.

It's a simplistic tool with huge possibilities. I aim to be rather simple for the first iteration out of the door.

Speaking of the first iteration out of the door - I ended up copying/pasting a LOT of the internal code in the Adobe AIR client to finish it. Having a local backup database in case of internet connection failure was a "last minute" additional "requirement" that I hadn't anticipated in the planning process, so I had to hack a lot of ugly stuff in to the code - and to finish it without a lot of it being rewritten, well... Thankfully, I do intend to upgrade it, as I left out one (actually two) important features.

1. Remote upgrade.
(2. Specify end-points to receive dynamic setup info.)

I left out the upgrade script because (1)I've heard it's a resource hog, and is a pain to work with, and...(2)I know I'll need to manually upgrade our library's PCs at the next version because of DeepFreeze anyway. (I left out the portion of the settings because I'm not yet ready to release the code as open source just yet, so I can hard code that stuff for now.)

I will post pictures of both the client and the web tool after it's been in use for a couple weeks - I'd gather that'll probably be about a month from now (I can't start work on the web panel until Monday, and it'll probably take me about two weeks to finish).

Here I was, minding my own business, when I try to insert the current time in to a SQLite database via Adobe AIR. AIR has some internal mechanisms to actually give you an error if you don't insert the proper type in to an SQLite database (unlike SQLite itself, which will let you put anything in to any type of field - it just casts the values, from what I understand). Well, it sort of inserted the current time. SQLite has a shorthand insertion value for DATETIME fields when inserting the string value of "now". It will insert (you guessed it), the current time as it is NOW. When you view the "current" time, however...let's say the current date and time is: Tuesday, September 8th 2009 at 2:46:21pm EDT (that is 4 hours behind GMT). So when viewing the "current" time that you just inserted in to your SQLite database, you will see:
Tue Sep 8 2009 18:46:21 -0400 GMT

...what?

Okay, I get the military time, but, wouldn't that be 6pm? It's showing my timezone as EDT (-4 hours from GMT), so why is it listing 4 hours difference?!

Well, the answer basically is: I have no idea why it lists the timezone the way it does. If you want to insert your local time, you need to use the following:
INSERT INTO `test_table` (`any_date`) VALUES (DATETIME("now", "localtime"));

The shorthand property of "now" can also be written as "DATETIME('now')", and the DATETIME method (in SQLite) apparently takes more than one parameter, and had I known what my issue was in the first place, it would have been easier to track down. You can find (as of September 8, 2009) SQLite's date and time methods at the following URL: SQLite Date and Time Functions.

Hopefully someone else doing a web search while grasping for straws might find this, and it'll help them out, rather than have them waste the better part of a day scratching their head, thinking of a workaround but knowing they must be missing something simple.