ebyblog - libraries, technology and farming

Cross-posted from the AADL Devblog

The AADL Summer Game has an online shop where earned points can be spent on awesome schwag. To try to make it easy for the volunteers and staff to fulfill the orders we took advantage of some of the infrastructure we already had in place for other parts of the site.

As some know our hold notices currently go through a script that sends an email along with printing a custom label to a label printer, that is used for identification on our hold shelves. We reused this process to print a custom pickup/order label everytime an order comes through the ubercart game shop by hooking into the payment process. This leaves a spool of order labels that those doing fulfillment can pick up throughout the week and start filling. A spool that has been increasing dramatically in length recently.

As orders are filled the barcode on the custom label is scanned, which sets the order as fulfilled and adds a notification job to a redis queue. Players have the option of getting SMS notifications (sent through Twilio) or email. Shop keepers can also cancel orders if need be which refunds the points to the player account.

After the items are delivered to the branch destinations on Friday morning, a script goes through the jobs on the redis queue and sends the notifications letting players know their items are ready for pickup.

More background info on our Summer Game is in the works and keep that order spool growing!

Cross-posted from the AADL Devblog and written by our lead developer ejk

The 2011 Summer Game has brought big changes to the way we play here at the library. In addition to the "classic" summer reading game, players can earn points for writing reviews, adding comments and finding game codes at events and locations. Players also earn badges for special accomplishments. We just passed player ID #4000 and we still have weeks to go for even more players to join and earn points and prizes. The pieces that make up the Summer Game are diverse but by adding custom code to the solid foundations provided by these open tools we've been able to concentrate our efforts on adding new content and functionality rather than chasing bugs and putting out fires. Here's some of the tools and technologies that make up the Summer Game:

Drupal: The aadl.org websites run on the Drupal Content Management System. In addition to giving us a framework for writing blogs, creating user accounts and writing comments, it has a extensive API which allows us to leverage those pieces to add our own functionality. A drupal module for Summer Game was created that keeps track of player data, lets players add points through multiple activities, and displays a leaderboard. Players are attached to user accounts, and a simple function which could be placed in any code that runs the site allows us to award points for any website action. The summer game module is available through github: Summer Game module.

Ubercart: The Summer Game shop runs on this Drupal module that provides a simple shop interface for your Drupal site. Again, it has a extensive API which allowed us to use Summer Game points as a custom payment type. We also added a hook into the order process to send data to our custom label printing function, printing labels for the items at the time of order, and creating custom order emails. By leveraging the ubercart module, we were able to create a complete online shop and order fulfillment process in about 3 weeks, start to finish.

FPDF/FPDI: The core of the Summer Game is still the "classic" summer reading game, where you read books, fill out a score card, bring it to a library location and get a prize. This year we decided to track all our score cards with unique identifier numbers to see where and when people picked them up. We also wanted to allow players to print their score cards on demand as PDF files. The FPDF library allows us to dynamically create each score card PDF with a unique number. We also use the FPDF library to create easily-printable Game Code signs so staff can post a points-accruing Game Code at their events. The FPDI plugin for FPDF allows you to import an existing PDF as a template for your FPDF document.

Redis: Redis provides a persistent data store, similar to a database, but as key-value pairs rather than tables. We used Redis to store the score card identification numbers because it allowed us to do an atomic increment operation. We don't want two people printing score cards at the same time with the same ID number. In a single call we can get the next number in the sequence without worrying about holding other people up or both grabbing the same number. We also use Redis to store our shop email notification data in a processing queue which allows us to delay sending them out. We currently use the Redisent PHP library to talk to our Redis install.

Twilio: Twilio provides an easy API to interact with users via cell phone text messages. By creating a simple drupal module that utilizes the Twilio API we allowed players at an event to easily sign up for the game and start earning points using their cell phones. We also use twilio to send prize pickup reminder text messages when we fill orders from the Summer Game shop.

AADL is happy to share source code. View the rest of our shared code on our github page. Code monkeys not included. Your mileage may vary.

Prompting a browser to download rather than open something is rather common. You can do so from your favorite web programming language by adding the Content-Disposition attachment header when sending the data back.

This can come in handy if you want to use the same file for both web serving and download. Say a MP4 movie file. It makes it a little easier than the right-click save-as method.

If you want to remove the overhead of serving a file through a script you can also accomplish this straight in nginx. In the example below the file can be served both ways.

Served normally: http://example.org/mymovie.mp4

Served as attachment: http://example.org/mymovie.mp4?name=mymovie.mp4

This would go inside the location declaration:

  if ($args){
    add_header X-Content-Type-Options nosniff;
    add_header Content-Type "application/octet-stream";
    add_header Content-Disposition 'attachment; filename="$arg_name"';
  }

This also gives you the option of having the downloaded file be named differently. So if you have directory structure like /2011/mysong.mp3 you could have it save as 2011-mysong.mp3.

This also lets you take advantage of other nginx features like X-accel when serving or streaming the files.

Cross-posted from the AADL Devblog

Over the past few years we've taken advantage of having a catalog we can tweak and change. Recently as we've been able to store more data outside of the ILS we have gone the route of making our catalog one of main development platforms.

However, staff still had to go back to the staff client for the ILS for some functions. While many of those functions don't really make sense as part of the catalog, requesting items for other patrons was one that did. Having full control over the search and what fields are indexed created a back and forth between the client and website for finding things.

We just rolled out the feature for staff so they can request for others via the public catalog (if they have the permission). Its a feature I'd recommend looking at implementing if you are working on an opac replacement or other catalog like feature.

Just one more step moving more non-inventory things to the drupal/sopac side of things rather than the ILS.

Google just released a research paper entitled Availability in Globally Distributed Storage Systems. It is available for download in PDF format (14 pages). From the abstract:

We characterize the availability properties of cloud storage systems based on an extensive one year study of Google's main storage infrastructure and present statistical models that enable further insight into the impact of multiple design choices, such as data placement and replication strategies. With these models we compare data availability under a variety of system parameters given the real patterns of failures observed in our fleet.

It is worth taking a quick browse of. The paper focuses on data availability (online versus data backup/integrity) and includes things like planned downtime. Metrics are given to assist in system design. Some highlights:

• GFS waits around 15 minutes before trying recovery process as that is more than the average time a node might return itself to normal operation (reboot, etc)
• Many events are rack and multi-rack level (network, rolling reboots, etc)
• Failures tend to happen in bursts
• Important to take the above two into account when planning replication schemes
• Reboot time is important
• Storing data across data centers reduces data unavailability by many orders of magnitude compared to having the same number of replicas in a single data center