Hackszine.com

Most recent posts: page 3 of 6 1 2 3 4 5 6
Browse the complete archive by category or month.

July 15, 2008

When to denormalize

There's been a bit of a database religious war on Dare Obasanjo and Jeff Atwood's blogs, all on the subject of database normalization: when to normalize, when not to, and the performance and data integrity issues that underly the decision.

Here's the root of the argument. What we've all been taught regarding database design is irrelevant if the design can't deliver the necessary performance results.

The 3rd normal form helps to ensure that the relationships in your DB reflect reality, that you don't have duplicate data, that the zero to many relationships in your system can accommodate any potential scenario, and that space isn't wasted and reserved for data that isn't explicitly being used. The downside is that a single object within the system may span many tables and, as your dataset grows large, the joins and/or multiple selects required to extract entities from the system begins to impact the system's performance.

By denormalizing, you can compromise and pull some of those relationships back into the parent table. You might decide, for instance, that a user can have only 3 phone numbers, 1 work address, and 1 home address. In doing so, you've met the requirements of the common scenario and removed the need to join to separate address or contact number tables. This isn't an uncommon compromise. Just look at the contacts table in your average cell phone to see it in action.

Jeff writes:

Both solutions have their pros and cons. So let me put the question to you: which is better -- a normalized database, or a denormalized database?

Trick question! The answer is that it doesn't matter! Until you have millions and millions of rows of data, that is. Everything is fast for small n.

So for large n, what's the solution? In my personal experience, you can usually have it both ways.

Design your database to 3NF from the beginning to ensure data integrity and to allow room for growth, additional relationships, and the sanity of future querying and indexing. Only when you find there are performance problems do you need to think about optimizing. Usually this can be accomplished through smarter querying. When it cannot, you derive a denormalized data set from the normalized source. This can be as simple as an extra field in the parent table that derives sort information on inserts, or it can be a full-blown object cache table that's updated from the official source at some regular interval or when an important even occurs.

Read the discussions and share your comments. To me, the big takeaway is that there's no one solution that will fit every real world problem. Ultimately, your final design has to reflect the unique needs of the problem that is being solved.

When Not to Normalize your SQL Database
Maybe Normalizing Isn't Normal

July 14, 2008

LEGO Wall-E

NXT Mindstorms hacker BlueToothKiwi created a working Wall-E trash collector robot and uploaded build instructions to the NXTLog.

To mark the release of the film, the official web site has a 'Build your own robot' section where you get to choose the looks / behavior / mobility etc.
And of course, if you got a NXT - you dont need to go to a web site to design a virtual robot. You can of course build your own real Wall-e! Well almost!!

BlueToothKiwi's bot may have won an NXTLog building challenge, but there are some other great Wall-e robots on the NXTLog site worth checking out. From the look-and-feel department, Joe Meno's Flickr photos of his bot bear a striking resemblance to Earth's last robot.

Last is this humble little Wall-E, based on the original Mindstorms.

I'm pretty sure I need to build about 10 of these things and scatter them around the office.

Making your own Wall-e with NXT [thanks, Patti]
Instructions on NXTLog
Other Wall-e bots at the LEGO NXTLog
Joe Meno's LEGO Wall-e on Flickr

July 13, 2008

Find and Grep 101

Find and Grep are perhaps the most used command line tools for the Linux user or administrator. Terse but powerful, these two commands will allow you to search through files and their contents by almost any imaginable attribute or filter criteria: file name, date modified, occurrence of the some specific word in a file, etc. Combined with a couple of other standard unix utilities, you can automate and process modifications over a number of files that match your search.

Here are two blog posts by Eric Wendelin which nicely illustrate the basics of these two commands:

Find is a Beautiful Tool
Grep is a Beautiful Tool

There are a number of other great unix utilities for file search, but knowing how to use find and grep is fundamental, as these two utilities can be found on the most basic build of every unix-like machine you come across.

Got a favorite command line hack that uses find or grep? Drop it on us in the comments.

July 12, 2008

Cruel Super Mario World hack

Kaizo Mario is a homebrew level for Super Mario World that's equal parts evil and genius. My tolerance for frustration isn't nearly high enough to be able to handle this, but for those of you looking for a gaming challenge, there's a link to the ROM below.

You can make your own custom levels for Super Mario World with a graphical level editor called Lunar Magic. You'll need a Windows PC and the original SNES Super Mario World ROM to use it. If you come up with anything you'd like to share, please add it to the comments.

Kaizo Mario Download
Lunar Magic: Super Mario Wolrd Level Editor

July 11, 2008

Reverse autocomplete

Traditional autocomplete is such a powerful tool that it's managed to work its way into most desktop and a significant number of web applications. Type a URL into your browser, and the address bar will offer suggestions for all of the URLs in your browsing history that begin with the text that precedes your cursor. Autocomplete works well, but it could be better.

László Kozma brought up a problem that tends to crop up in a number of scenarios: if you move the cursor back to correct or change part of the entry string, traditional autocomplete completely ignores any context to the right of the cursor. In some applications, this means everything to the right of the cursor is ignored and overwritten instead of being part of the search. In Safari, if you type anything between "www." and ".com", autocomplete fails entirely, offering no results unless you clear out everything to the right of the cursor.

One solution, which László termed "reverse autocomplete", is to split the string at the cursor position and attempt to find matches for both anything after the first half and before the second half. Any matches that show up in both sets are the final autocomplete suggestions. The result is that if I type in "www.h|.com" (where my cursor position is represented by the "|" character) the smarter autocomplete might return "www.hackszine.com" as a suggestion but omit from telling me about "www.h-is-aitch.org".

You can take it one step further and also match against the beginning and ends of the entire string. This solves a really common problem that I run into regularly when searching through a large contact list, say the typical corporate email system. If you don't know the correct spelling of someone's name, or if you only can recall initials, you can fill in the parts you know. The system would be smart enough to turn "J|S" into "Jason Striegel" instead of forcing you to page through a huge list of J names.

Reverse autocomplete

July 10, 2008

Mapstraction - map abstraction API for Javascript

Mapstraction is an abstracted Javascript mapping API that can make use of Google Maps, Microsoft Virtual Earth, Yahoo Maps and Mapquest. Instead of deciding on a particular mapping provider, you can build your web application with Mapstraction and easily switch to a different service by changing a single line of code. From the Mapstraction site:

Mapstraction additionally fills some holes each provider's current offerings (taking advantage of existing open source solutions where possible) to normalise the feature set across platforms. In the future, Mapstraction will also talk to OpenStreetMap for people who want to build maps without restrictions on derived works.
Features

Support for 9 major mapping providers
Point, Line, Polygon support
Image overlay
GeoRSS and KML feed import
Geocoding of addresses
Driving directions

There's an introductory walkthrough on Webmonkey that shows you how to do the basics like instantiating a map with various providers and adding markers with the abstracted API. The Mapstraction web site also has demos for geocoding, drawing polygons, and swapping map tiles. The API appears to cover all the bases. I can't think of any reason to directly use a specific map provider instead of this.

Mapstraction
WebMonkey Mapstraction Tutorial

July 9, 2008

Maglite LASER burnination

Desertfoxx sent us a tip to a classic KipKay howto video. It's pretty easy to swap out the diode in a laser pointer or similar laser housing with the diode from an old DVD burner. Put the whole deal into a mini Maglite and you've got yourself a nice little handheld fire starter, perfect for lighting matches or popping dark colored balloons from across the room. Like all lasers, it's also excellent for quickly blinding people, so watch where you point the thing.

If you haven't caught KipKay before, he's been doing some fun weekend project video podcasts for MAKE.

How To Make A Burning Laser Flashlight

July 8, 2008

3D Studio Max motion capture with a Wii Nunchuck

By passing Nunchuck data to a PC via an Arduino, Melka figured out a way to convert the accelerometer output into a MIDI stream that can be read directly by 3D Studio Max's motion capture engine:

Here's my setup under windows :
Arduino using a WiiChuck adapter from todbot (thanks kurt ^^) and the WiiChuck library from Tim Hirzel
Data sent to Processing via serial connection and translated to MIDI CC messages using the proMIDI library by Christian Riekoff
MIDI output from processing sent to midiYoke
midiYoke sends this data to Ableton Live
Ableton re-sends the CC messages to midiYoke
Using Float Motion Capture controllers on 3D Studio Max to rotate the objects according to the pitch and roll of the wiichuck

It's a little complicated, but from the looks of the video the payoff is worth it. You could adapt this to use data from a number of accelerometers, or turn other measurement data into a MIDI stream that can be used by any application, 3DS or otherwise.

Arduino to 3D Studio Max [via MAKE]

July 7, 2008

PlaceSpotting - Google Maps geo quiz

Martin Fussen tipped us off to PlaceSpotting, a user-contributed geo quiz map mashup. The idea is to create puzzles for your friends to solve by picking an obscure landmark and supplying them with a few hints. Your friends can then zoom around on the map to find the location. If they position the map at the right zoom level and over the location the puzzle is solved. I have to say this is a pretty fun way to learn a bit about the world, especially if you're in to puzzles and treasure hunts.

You can search and browse a large library of entries that other people have created. There are a significant number of entries in German, and many of the landmarks are within Europe, but there's nothing stopping you from dropping a few landmarks near the place you call home.

PlaceSpotting

July 6, 2008

KidWash sprinkler toy

Just because it's hot doesn't mean the kids have to stay indoors in the A/C. There are a number of worthwhile summer projects, but the KidWash looks like it has a particularly high fun/effort ratio. A trip to the hardware store for some PVC and mister jets and you can give the Wii a run for its money next weekend.

I headed down to the PVC section of the local home improvement store to pick up supplies. While browsing the adjacent sections for interesting stuff I noticed the micro-irrigation section and inspiration struck: KidWash with mister jets!
The modification worked great. We turned it on and kids from up and down the block started showing up to help with the testing. It's a lot of fun on foot, but my kids also get a blast out of riding their bikes through it.

This would be great to combine with a DIY visqueen slip and slide.

KidWash 2 : PVC Sprinkler Water Toy

July 5, 2008

Crawling AJAX

Traditionally, a web spider system is tasked with connecting to a server, pulling down the HTML document, scanning the document for anchor links to other HTTP URLs and repeating the same process on all of the discovered URLs. Each URL represents a different state of the traditional web site. In an AJAX application, much of the page content isn't contained in the HTML document, but is dynamically inserted by Javascript during page load. Furthermore, anchor links can trigger javascript events instead of pointing to other documents. The state of the application is defined by the series of Javascript events that were triggered after page load. The result is that the traditional spider is only able to see a small fraction of the site's content and is unable to index any of the application's state information.

So how do we go about fixing the problem?

Crawl AJAX Like A Human Would
To crawl AJAX, the spider needs to understand more about a page than just its HTML. It needs to be able to understand the structure of the document as well as the Javascript that manipulates it. To be able to investigate the deeper state of an application, the crawling process also needs to be able to recognize and execute events within the document to simulate the paths that might be taken by a real user.

Shreeraj Shah's paper, Crawling Ajax-driven Web 2.0 Applications, does a nice job of describing the "event-driven" approach to web crawling. It's about creating a smarter class of web crawling software which is able to retrieve, execute, and parse dynamic, Javascript-driven DOM content, much like a human would operate a full-featured web browser.

The "protocol-driven" approach does not work when the crawler comes across an Ajax embedded page. This is because all target resources are part of JavaScript code and are embedded in the DOM context. It is important to both understand and trigger this DOM-based activity. In the process, this has lead to another approach called "event-driven" crawling. It has following three key components
Javascript analysis and interpretation with linking to Ajax
DOM event handling and dispatching
Dynamic DOM content extraction

The Necessary Tools
The easiest way to implement an AJAX-enabled, event-driven crawler is to use a modern browser as the underlying platform. There are a couple of tools available, namely Watir and Crowbar, that will allow you to control Firefox or IE from code, allowing you to extract page data after it has processed any Javascript.

Watir is a library that enables browser automation using Ruby. It was originally built for IE, but it's been ported to both Firefox and Safari as well. The Watir API allows you to launch a browser process and then directly extract and click on anchor links from your Ruby application. This application alone makes me want to get more familiar with Ruby.

Crowbar is another interesting tool which uses a headless version of Firefox to render and parse web content. What's cool is that it provides a web server interface to the browser, so you can issue simple GET or POST requests from any language and then scrape the results as needed. This lets you interact with the browser from even simple command line scripts, using curl or wget.

Which tool you use depends on the needs of your crawler. Crowbar has the benefit of being language agnostic and simple to integrate into a traditional crawler design to extract page information that would only be present after a page has completed loading. Watir, on the other hand, gives you deeper, interactive access to the browser, allowing you to trigger subsequent Javascript events. The downside is that the logic behind a crawler that can dig deep into application state is quite a bit more complicated, and with Watir you are tied to Ruby which may or may not be your cup of tea.

Crowbar - server-side headless Firefox
Watir - browser remote control in Ruby
Crawling Ajax-driven Web 2.0 Applications (PDF)

July 4, 2008

Direct video manipulation interface

Direct manipulation of video is one of the more uncanny HCI concepts I've ever seen. Instead of manipulating time with a traditional scrubber bar, the user can drag objects in the video across their path of movement. Nothing in the video actually changes, but the perception is that you can directly manipulate the objects in the video stream by pulling them around through time.

There's a Windows application called DimP which implements this interface. When you hover over a movable object in the video, a light path appears that emphasizes the object's motion curve, which you can then move the object across. From the DimP website:

So what's being manipulated, exactly? Both the video content (e.g., the things you see moving in the video) and the "tape head". When using DimP, the user directly manipulates the video content and indirectly manipulates the tape head. When using the seeker bar, the user directly manipulates the tape head and indirectly manipulates the video content.

The video above describes how DimP works in a bit more detail, showing a few different video scenarios where direct manipulation really shines. It's intuitive and bizarre at the same time. If the universe is completely deterministic, I can't help but think this is what time travel must look like.

DimP - A Direct Manipulation Video Player
DRAGON - Direct Manipulation Interface Demo for OS X

July 3, 2008

Make a record player out of LEGO

A paper cup, a sewing needle, and a simple LEGO Mindstorms robot make for a nice little record player. This might be a fun project to work on with the kids this 4th of July weekend. You know they've been wondering about the boxes of vinyl frisbees in the basement. Aside from anything else they might be learning, it's a good opportunity to slip in some information about how music used to be groovy.

NXT Record Player Instructions

July 2, 2008

Algorithm Ink and ContextFree.js - generative art with Javascript

Currently Firefox, Safari and Opera have reasonable support for the Canvas HTML element. With IE the only major remaining laggard, there are a lot of people starting to experiment with Javascript's new graphical capabilities. I mentioned John Resig's Processing.js library in May, and now Aza Raskin has released the ContextFree.js library, which brings another generative drawing language to web-standards software development.

Besides being pretty, why is ContextFree.js interesting? Because it shows the power of Open web technologies for making graphically-enabled, compelling interaction. The true power of the web revolves around anyone being able to dive in, see what someone else has done, and expand upon it. Canvas lowers the cost of entry to creating graphical mashups and other dynamic, graphical content. It also shows the progress the web has made: a year ago, this demo would not have been possible. Canvas wasn't ready, and Javascript interpreters weren't fast enough. Looking at the qualitative difference in speed from Firefox 2 to Firefox 3 indicates the amazing and substantial progress made towards speeding up Javascript since the last major browser release cycle.

ContextFree.js is a Javascript port of the Context Free open source generative art application by Chris Coyne. It basically defines an extremely simple grammar that is designed to generate rule-based artwork with very few lines of code.

Aza has also released the Algorithm Ink website, which uses ContextFree.js to create an open source art gallery. Using Algorithm Ink, you can load, tweak, and share generative art through a web interface. When you see something you like, you can view the source for the artwork and use it in your own creations. Very cool.

ContextFree.js & Algorithm Ink: Making Art with Javascript
ContextFree.js at Google Code
Algorithm Ink
The original Context Free by Chris Coyne
Drawing Graphics with Canvas
Processing.js - visualization library for Javascript

July 1, 2008

Swurl - scrapbook your digital life

Ryan Sit sent in a link to Swurl, a Web 2.0 application/startup he's been working on that attempts to collect all of the artifacts of your digital life and assemble them into a sensible whole. You could call it an aggregation tool—which, deep down, it is—but it does a little more than other things I've seen. Namely, it does a really nice job of taking in a wide range of relationship oriented services and turning the their inherent lock-in inside-out.

You just do your normal stuff online, Digg stuff, Delicious stuff, favorite Youtube videos, Twitter, rent videos on Netflix, bookmark songs on Last.fm, post photos on Flickr, etc. And Swurl brings it all together in a really deep way. We also enhance data, adding trailers to movies you rent and Lyrics to songs you bookmark.

All of these services get pulled together into a blog format and discussion and commenting can take place around any item. I think the idea is to help centralize the conversation instead of having it scattered about your different networks. Depending how busy your various online activities are, this may help you or it may just add yet another place to track comments.

What I like most, and the real hack from a Web 2.0 startup perspective, is that they programmed this to encourage you to maintain friends and use features across a number of social networks and easily traverse the relationships and data in all of them from a single location. Flikr is good for storing photos. Del.icio.us is great for bookmarking. Facebook is awesome for tracking friends. Swurl aggregates the relationships from all of the networks you inform it about and it recognizes when those users are also Swurl members. This encourages network growth, but isn't so greedy that it forces you to work only inside the Swurl fence.

Swurl [via Ryan]

June 30, 2008

Add-Art - turn the adverweb into an art gallery

Add-Art is an ad blocking extension for Firefox, but it's a bit different from other ad blocking hacks. Instead of simply removing ads, leaving a score of empty holes in ad-laden sites, Add-Art replaces all those nasty skyscrapers and leaderboards with a selection of images from a contemporary artist. The image above, for instance, is a cropped banner from Add-Art's current show, Hotel Vue des Alpes by Monica Studer and Christoph van den Berg.

In addition to turning your browser into an art gallery, blocking ads might actually speed up your browsing experience a bit. The Add-Art images are distributed with the plugin and are loaded locally from your machine, so there's no download involved for any of the ad units. Just think of all those bytes you could be saving while viewing lovely mountain scenery instead of dancing mortgage dudes.

Add-Art Firefox Plugin
Hotel Vue des Alpes

Google Apps Hacks Slashdotted today!

Yay! Philipp Lenssen's Google Apps Hacks just got reviewed on Slashdot by JR Peck, who had some kind words for the book:

Well, this book is an excellent introduction to Google's many on-line applications. I use many already and still learned of a couple new ones when I read this book. It also does bring all that instruction into one place, and provides a very user-friendly style of instruction. There is also a very nice feature, 8 sections that take the reader "Beyond Google...". Each of these sections informs the reader about alternative software that provides similar functionality to the Google software described in the preceding chapter. This is really a great resource and an unexpected bonus for anyone who reads the book.

Slashdot | Google Apps Hacks

(In the review, JR raises a good point about the definition of a Hack. We use the definition "A non-obvious solution to an interesting problem", although JR leans toward some other definitions mentioned in the review).

From the Maker Shed:

Google Apps Hacks by Philipp Lenssen - all about Google Docs & Spreadsheets, Gmail, Google Calendar, Google Presentations, and more!
Price: $29.99
Buy it in the Maker Shed

June 29, 2008

Objective-J and Cappuccino: Cocoa for the web

There's a neat article over at Ars Technica that takes a look at 280 North's 3-person development team, their recent release of a keynote-like web application called 280 Slides, and the framework that they constructed to make the application. Some time before 280 Slides was developed, the team created an Objective-C-like superset of the Javascript language called Objective-J which implements message passing and a Smalltalk-style syntax. Building on that, they re-implemented much of the Cocoa framework in Objective-J, allowing Cocoa-style applications to be developed that will run natively in the browser.

With Objective-J and Cappuccino, you don't create applications with a mixture of HTML, CSS and Javascript. Instead, apps are written entirely in Objective-J, following a development model that's similar to creating desktop applications for OS X. From the article:

"Cappuccino is a re-implementation of Cocoa in Objective-J, which means we reimplemented AppKit, Foundation, CoreGraphics, and parts of CoreAnimation," Boucher told us. With it, developers familiar with desktop GUI applications can create a rich, desktop-like web app with the same relative ease Cocoa programmers can create a rich desktop app. "Coming from a background of desktop programming, and Cocoa in particular, we realized how much harder building a web application was. So we wanted to try to make things just a little bit easier."

280 Slides stands on its own as a powerful web-based presentation tool. It's simplified when compared to its desktop cousins, but it does exactly what most people need in a presentation tool, and it can export to PPT for fine tuning and end-presentation use. The fact that its development inspired the creation of a new language as well as bringing a desktop application development framework to the web just blows my mind.

Objective-J will be released to the public soon as an open source project. On the one hand, I'm not sure if I'm comfortable moving away from the web development technologies I've grown accustomed to. On the other, it would be cool to work in a language that allows you to seamlessly port your applications between the desktop and the web.

If you have experience with both Cocoa and AJAX development, please share your thoughts in the comment area.

Cocoa on the web: 280 North, Objective-J, and Cappuccino [via Slashdot]
280 North Blog
280 Slides
Objective-J.org

June 28, 2008

JSVI - Javascript Vi for web forms

There are so many WYSIWYG editors for html forms, it's about time someone put together a decent web version of the Vi editor. JSVI is written in Javascript and seems to be fairly cross-browser compatible. With a few tweaks to your code, you can now make any textarea behave exactly like Vi. How cool is that?

To use it, download the GPLed vi.js file from the link below, include it in your html document, and then add an onfocus="editor(this);" to any textarea that you want vi enabled. If your textarea contains any preset information, the text will be loaded into the editor when the field is clicked. Saving the file with :wq will exit the editor and store the changes back in the input field.

Somewhere there must be a unix blog or forum that will adopt this for their submission and comment fields. Shame on them if they don't. That is, unless they are waiting for the emacs version.

JSVI - VI in JavaScript (demo)
Download vi.js

June 27, 2008

Arduino VGA demo

Sebastian Tomczak has been playing around with controlling VGA output from an Arduino. Using Max/MSP to process audio and send data to the Arduino and a standard VGA output to send sync data to the monitor, he's been able to put together some cool video effects.

In these examples, i am simply using three pins to control the RGB lines. However, the Arduino is not generating horizontal or vertical sync - this is generated by a computer.

You can grab the Max patch and Arduino code from Sebastian's post. I dig the effect - it reminds me of the cool C4 demos kids used to make.

Hacking VGA lines with Arduino