ebyblog

Summer Game Prize Fulfillment Workflow

2011-08-10T19:00:00+00:00

Cross-posted from the AADL Devblog

The AADL Summer Game has an online shop where earned points can be spent on awesome schwag. To try to make it easy for the volunteers and staff to fulfill the orders we took advantage of some of the infrastructure we already had in place for other parts of the site.

As some know our hold notices currently go through a script that sends an email along with printing a custom label to a label printer, that is used for identification on our hold shelves. We reused this process to print a custom pickup/order label everytime an order comes through the ubercart game shop by hooking into the payment process. This leaves a spool of order labels that those doing fulfillment can pick up throughout the week and start filling. A spool that has been increasing dramatically in length recently.

As orders are filled the barcode on the custom label is scanned, which sets the order as fulfilled and adds a notification job to a redis queue. Players have the option of getting SMS notifications (sent through Twilio) or email. Shop keepers can also cancel orders if need be which refunds the points to the player account.

After the items are delivered to the branch destinations on Friday morning, a script goes through the jobs on the redis queue and sends the notifications letting players know their items are ready for pickup.

More background info on our Summer Game is in the works and keep that order spool growing!

Under the Hood of the AADL Summer Game

2011-08-10T18:00:00+00:00

Cross-posted from the AADL Devblog and written by our lead developer ejk

The 2011 Summer Game has brought big changes to the way we play here at the library. In addition to the "classic" summer reading game, players can earn points for writing reviews, adding comments and finding game codes at events and locations. Players also earn badges for special accomplishments. We just passed player ID #4000 and we still have weeks to go for even more players to join and earn points and prizes. The pieces that make up the Summer Game are diverse but by adding custom code to the solid foundations provided by these open tools we've been able to concentrate our efforts on adding new content and functionality rather than chasing bugs and putting out fires. Here's some of the tools and technologies that make up the Summer Game:

Drupal: The aadl.org websites run on the Drupal Content Management System. In addition to giving us a framework for writing blogs, creating user accounts and writing comments, it has a extensive API which allows us to leverage those pieces to add our own functionality. A drupal module for Summer Game was created that keeps track of player data, lets players add points through multiple activities, and displays a leaderboard. Players are attached to user accounts, and a simple function which could be placed in any code that runs the site allows us to award points for any website action. The summer game module is available through github: Summer Game module.

Ubercart: The Summer Game shop runs on this Drupal module that provides a simple shop interface for your Drupal site. Again, it has a extensive API which allowed us to use Summer Game points as a custom payment type. We also added a hook into the order process to send data to our custom label printing function, printing labels for the items at the time of order, and creating custom order emails. By leveraging the ubercart module, we were able to create a complete online shop and order fulfillment process in about 3 weeks, start to finish.

FPDF/FPDI: The core of the Summer Game is still the "classic" summer reading game, where you read books, fill out a score card, bring it to a library location and get a prize. This year we decided to track all our score cards with unique identifier numbers to see where and when people picked them up. We also wanted to allow players to print their score cards on demand as PDF files. The FPDF library allows us to dynamically create each score card PDF with a unique number. We also use the FPDF library to create easily-printable Game Code signs so staff can post a points-accruing Game Code at their events. The FPDI plugin for FPDF allows you to import an existing PDF as a template for your FPDF document.

Redis: Redis provides a persistent data store, similar to a database, but as key-value pairs rather than tables. We used Redis to store the score card identification numbers because it allowed us to do an atomic increment operation. We don't want two people printing score cards at the same time with the same ID number. In a single call we can get the next number in the sequence without worrying about holding other people up or both grabbing the same number. We also use Redis to store our shop email notification data in a processing queue which allows us to delay sending them out. We currently use the Redisent PHP library to talk to our Redis install.

Twilio: Twilio provides an easy API to interact with users via cell phone text messages. By creating a simple drupal module that utilizes the Twilio API we allowed players at an event to easily sign up for the game and start earning points using their cell phones. We also use twilio to send prize pickup reminder text messages when we fill orders from the Summer Game shop.

AADL is happy to share source code. View the rest of our shared code on our github page. Code monkeys not included. Your mileage may vary.

Forcing Download with nginx

2011-04-06T18:00:00+00:00

Prompting a browser to download rather than open something is rather common. You can do so from your favorite web programming language by adding the Content-Disposition attachment header when sending the data back.

This can come in handy if you want to use the same file for both web serving and download. Say a MP4 movie file. It makes it a little easier than the right-click save-as method.

If you want to remove the overhead of serving a file through a script you can also accomplish this straight in nginx. In the example below the file can be served both ways.

Served normally: http://example.org/mymovie.mp4

Served as attachment: http://example.org/mymovie.mp4?name=mymovie.mp4

This would go inside the location declaration:

  if ($args){
    add_header X-Content-Type-Options nosniff;
    add_header Content-Type "application/octet-stream";
    add_header Content-Disposition 'attachment; filename="$arg_name"';
  }

This also gives you the option of having the downloaded file be named differently. So if you have directory structure like /2011/mysong.mp3 you could have it save as 2011-mysong.mp3.

This also lets you take advantage of other nginx features like X-accel when serving or streaming the files.

Staff Requests in SOPAC

2011-04-05T18:00:00+00:00

Cross-posted from the AADL Devblog

Over the past few years we've taken advantage of having a catalog we can tweak and change. Recently as we've been able to store more data outside of the ILS we have gone the route of making our catalog one of main development platforms.

However, staff still had to go back to the staff client for the ILS for some functions. While many of those functions don't really make sense as part of the catalog, requesting items for other patrons was one that did. Having full control over the search and what fields are indexed created a back and forth between the client and website for finding things.

We just rolled out the feature for staff so they can request for others via the public catalog (if they have the permission). Its a feature I'd recommend looking at implementing if you are working on an opac replacement or other catalog like feature.

Just one more step moving more non-inventory things to the drupal/sopac side of things rather than the ILS.

Availability in Globally Distributed Storage Systems

2011-02-26T13:00:00+00:00

Google just released a research paper entitled Availability in Globally Distributed Storage Systems. It is available for download in PDF format (14 pages). From the abstract:

We characterize the availability properties of cloud storage systems based on an extensive one year study of Google's main storage infrastructure and present statistical models that enable further insight into the impact of multiple design choices, such as data placement and replication strategies. With these models we compare data availability under a variety of system parameters given the real patterns of failures observed in our fleet.

It is worth taking a quick browse of. The paper focuses on data availability (online versus data backup/integrity) and includes things like planned downtime. Metrics are given to assist in system design. Some highlights:

GFS waits around 15 minutes before trying recovery process as that is more than the average time a node might return itself to normal operation (reboot, etc)
Many events are rack and multi-rack level (network, rolling reboots, etc)
Failures tend to happen in bursts
Important to take the above two into account when planning replication schemes
Reboot time is important
Storing data across data centers reduces data unavailability by many orders of magnitude compared to having the same number of replicas in a single data center

Bagit Validation and CIJoe

2011-02-18T16:00:00+00:00

I've started playing around with bagit and some of the other curation microservices. One of the features involved is manifest files w/ checksums of all the data files in the bag. So far I'm still just getting the hang of it but decided to try cbeer's bagit gem and try hooking the validate function to cijoe integration server. With the bag in git repository it was easy to have cijoe run a simple script that returned zero when validated, or one with output if there was a problem. cijoe can use git hooks to notify via various methods (example one for email is given in repo).

While not robust it only took a minute to get the output displayed:

larger version

A few thoughts:

need to tweak cijoe to work on local repository instead of requiring an origin
keeping the validation script in the repository helps. should probably validate/manifest it as well

Overall a lot easier than I thought it would be and serves as a simple interface for seeing current status for those who like a webpage. cijoe also has jsonp and a simple http status api that allows other tools to easily look at build status so seems like it could be an easy tool to run all your tests in one place and let various things ping it.

IE8 Upgrade and Enhanced Security Stuck For Some Profiles

2010-04-05T00:00:00+00:00

Had an issue with IE upgrades in the recent past where after the upgrade multiple pre-existing profiles would state they were in "Enhanced Security Mode" despite that mode being disabled. New profiles would work fine and Administrator usually did as well.

From what I've been able to track down, MS hasn't figured out the exact circumstances that cause it. Fresh installs w/ upgrades before anyone logs in seem to work fine in my experience. Some profiles would also work fine despite being similar to ones that did.

Searches brought up multiple hacks to fix it. One is to make a login script that modifies the registry like so for every user:

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings]
"IEHardenIENoWarn"=dword:00000000

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap]
"IEHarden"=dword:00000000

The best solution I found, though, was to use GPO and force a setting for the Internet Zone. Forcing it to Medium or whatever level appears to get pass the enhanced security issues and things work like normal.

Under either User or Computer depending on how you have GPO setup you would want to enable and set a level for:

Administrative Templates>Windows Components>Internet Explorer>Internet Control Panel>Security Page>Internet Zone

Not ideal but it is the best I've found

If I Had A Heart

2009-12-05T00:00:00+00:00

If I Had A Heart from Fever Ray on Vimeo.

Unicode and Sphinx

2009-11-21T00:00:00+00:00

I'm not an expert with unicode and ran into various problems in getting Sphinx up and running with bib data for SOPAC2. Below is an overview of what I ended up doing to solve it. There are probably better ways of doing it and if so please share.

The Sphinx I'm referring to is the indexing/search program that works well with MySQL. The case below is using the stock searchd/indexer interface and not the mysql engine which is probably different.

Convert to UTF8 in MySQL

You can convert tables and all the fields within the table with the command (from pythoneer):

ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;

You could also do utf8_general_ci. From what I've picked up online is that the unicode_ci supports expansions/ligatures while the general doesn't which can affect sorting. However, the docs say the the tradeoff is speed in some operations. Both lack some character support versus the actual Unicode Collation Algorithm.

Configure Sphinx for Unicode

You will want to set your index definition to have:

charset_type = utf-8

While the docs mostly just mention that I highly recommend adding the following to your source definition:

sql_query_pre = SET CHARACTER_SET_RESULTS=utf8
sql_query_pre = SET NAMES utf8

While you might think that having your tables in utf8 collation, mysql set to default to it and php encoding in utf8 would be enough, if the client doesn't specify that it is doing utf8 mysql will store it in whatever encoding it thinks it is getting. I battled with php configs to try to ensure all machines modifying the tables were talking the same but you'll save yourself the trouble by defining this here and also in your scripts that write to the tables.

Modify your insertions to SET NAMES

As stated above I'd recommend doing a SET NAMES utf8 in front of your queries inserting data into mysql. This again may be poor setup on my part but it helps rule out different client configurations and probably makes it more portable. I found Gentoo by default seems to do fine, Debian hit or miss. Adding the SET made it work where ever I tried.

Some documentation stated I should wipe database if it previously had other encodings. I found this unnecessary and just inserted the data with UPDATE or REPLACE INTO and had no issues.

Set up character folding

While stop words and similar allow you to use external files for the list, it appears character folding needs the list to be in the config itself. At least I haven't been able to get an include file to work. I'd be interested in knowing otherwise. Depending on the languages you have you may want to generate a list of the codes present in your dataset. Otherwise you can find some tables to try out here:

Some reported that Sphinx can have issues with long lines in the config so you may need to break up the lines with \ characters. Your list should also include "0..9, a..z, A..Z->a..z" in it as you are overriding the default list.

Reindex

Another thing to note with Sphinx is that many changes to the sphinx.conf requires a restart of searchd. With the modification of the folding table you'll probably want to do this. Restart searchd and then do your indexer run for your indexes.

Try out some searches with keywords/characters and watch the query log. If the character folding is working then you should see the encoded characters in the query log. If they are missing or replaced by spaces then the characters aren't matching up with your folding table. Try to find the character, add it to your list, restart searchd and reindex. Sphinx defaults to not accepting characters that are not in the charset_table list. The query log is the easiest point to diagnose this at.

Web Display

The last step is of course to make sure your web pages are served in a proper encoding so browsers render it properly.

Again this isn't in depth but hopefully will help those messing with Sphinx.

Describing Everything - Open Web standards and classiﬁcation

2009-11-21T00:00:00+00:00

via @edsu

Describing Everything - Open Web standards and Classification

View more presentations from danbri.

Public Computing with Windows Servers and Linux Thin Clients - Hardware

2009-06-16T00:00:00+00:00

(also posted on the AADL devblog)

We recently overhauled the public computing setup at AADL, though it hasn't been rolled out at all the branches yet. It consists of a mix of linux hosted web management software, linux thin clients and windows terminal servers. It is a bit of a unique setup so figure I should share. Once we get it farther towards complete we'll probably release the code.

Some History

For some background the previous setup consisted of windows thinclients and windows servers running Citrix. The thin clients had published application sets that connected to a fairly basic server farm. The farm wasn't really setup well resulting in high loads and slow logins. A custom set of flash applications and php gateways administered it all.

As I researched options I decided I wanted to drop Citrix as I didn't really see it as needed and the licensing costs aren't exactly small. I also decided I would prefer as much be opensource as possible though we decided on keeping Windows for the public facing part for now.

I tested out quite a few public computing management software, lockdown software, cyber cafe software but didn't really find anything that did everything I wanted, kept it fairly simple, was flexible, etc. For reference here are a few things about our setup:

Patrons can use the computer as long as they want without interruption unless there are other patrons waiting for a computer. They have a minimum 30 minutes if there is a line, after which they are given 5, 2, 1 minute warnings.
They can be idle for a total of 10 minutes before being ended (given a warning after 5)
They swipe their card at an assigner first that gives them a station. These are spaced out then random

The New Thin Clients

The thin clients we settled on are HP Compaq t5735 Thin Clients. In bulk they ran under $250 each. The specs worked out for what we needed:

VESA mountable on both sides. Made it easy to securely mount to tables and even to the back of monitors
1 GB Flash, 256M RAM
VGA and DVI
Lots of USB ports including back, front and secure (within case)
Debian Linux preinstalled (stock plus some custom HP packages)

The thing I like most about the thin clients is the imaging process. The clients come with a HP programs called HPThinState which can write a bootable imager to a USB drive. Once the USB is imaged you can boot other machines from the USB and walk through a simple imaging process. The process is simple enough that front line staff now image machines themselves when needed (usually ext3 partition corruption after power failure). This has significantly cut time of IT staff and reduced how long a client is out of order for simple problems.

Another nice thing is that all changes are put on flash immediately, no special write/flashing software needed. You can apt-get upgrade and have the changes there on reboot with no further interaction.

Break Out Boxes

One of the biggest problems we had previously was USB ports being damaged on clients. We tried multiple things like hubs, etc but nothing really lasted. Our latest attempt is using a modular box that is meant to be inserted in a 5.25 cd bay on a tower. We put 2 usb extenders and a headphone jack extender in it and mount it to the table. This leaves an easily replaceable port and also lets us move the thin client out of harms way. So far this has worked great and we haven't had to replace a port yet. At about $10-12 it can't be beat.

Other Hardware

We have USB floppy drives available for use and may expand to other formats. The monitors are stock though we add on privacy screens. Each station is also equipped with a USB based barcode reader that is used for signin purposes.

We also have one of the thin clients VESA mounted to a monitor and barcode reader that acts as the assigner.

More information on the actual software / workflow behind it coming up in a second post. Overall though we've been really happy with the hardware described above. Really stable and much better priced.

Image versus XML Scanned Texts

2009-06-16T00:00:00+00:00

Another old post that is worth reading over at threepress about a case study in ebook presentation that has obvious application to scanning projects.

Most of the findings line up with experience but nice to see them quantified. Many preferred html versions for printability, layout, etc. Also covers workflow, costs.

ACLS Humanities E-Book XML Conversion Experiment: Report on Workflow, Costs, and User Preferences (pdf)

Gutter Garden

2009-06-16T00:00:00+00:00

An old article but still applicable for those with small spaces.

But our deck is on the wrong side of the house. Then an idea came to me that was a little unusual and might involve a little risk. The idea is essential this: Why not put rain gutters in rows along the wood siding on the sunny side of the house. It might look weird, but that was where all the heat, sun and protection from damage is best.

Amazon Treasures

2009-06-11T00:00:00+00:00

Revealing Design Treasures From The Amazon

Star Ratings

2009-02-14T00:00:00+00:00

From stevenf:

As soon as an app has been rated more than once, it becomes mathematically very unlikely that it will ever see a 1 or 5 star overall rating again. So, it's nearly pointless to have scales of 5 stars, 10 stars, or 100 stars, when all you really need is: "Liked it, Didn't Like It, and Neutral".

nude

2008-06-11T00:00:00+00:00

Big Ideas (don't get any) from James Houston on Vimeo.

AADL Library Camp 2008

2008-03-19T00:00:00+00:00

Just a reminder that the library camp is tomorrow, Thursday March 20th. More information is on libsuccess.org:

The next Library Camp will be held Thursday, March 20th, 2008, at the Downtown Branch of the Ann Arbor District Library, from 9:30 AM - 4:00 PM. The goal is to get a bunch of people together, and let them talk about whatever they'd like to talk about, within the rough purview of libraries and library technology. We'll start with all getting together with a blank schedule to talk about what we'd like to talk about and fill in a few slots on the agenda. We've got time for four sessions and room for at least 3 concurrent sessions, so there could easily be 10-12 different things to talk about during the day. We're working on providing breakfast; lunch is on your own, but there will surely be groups heading out to nearby places in downtown Ann Arbor. There is also a mid-range hotel near by (The Campus Inn, $200/night, orbitz has for $99/night), and a very cheap place a little further out that's not half bad (Lamp Post Inn, $45/night).

Txt Book Info

2008-03-13T00:00:00+00:00

I'm still digesting all the discussions and information from code4lib but I remembered a conversation I eavesdropped on. Casey reminded me it was between him and Adam Brin of Tri-College. The gist was that they had put a "txt this record to me" feature onto the catalog, without any real demand, which then took off to everyone's surprise. It was one of those "that would be cool" that then turned into a "how did we do without it" for some students. I believe they rapidly went up to 60 messages a day. You can see an example on The Selfish Gene. The feature itself sends the title and location info to your cellphone via SMS. They are using the free SMS email gateways I believe while Casey went with a pay service.

Most of my SMS plans allowed free incoming so I would have probably used this during my long college years. With the potential younger audience at public libraries I've heard discussion on whether kids racking up SMS charges using the catalog would be a good thing. Ed would also like free alternatives which probably wouldn't be hard for email if your already using the email gateways. I've already thought about all his options including adding it to our twitter script. When I revisit catwrap I'm going to look more into custom catalog experiences based on profile information such as twitter, email, phone, etc. With a little information in the profile it would be possible to slim down the display with just the applicable features (send to twitter, send to del.icio.us, etc).

Google Books Availability

2008-03-13T00:00:00+00:00

As you've probably read elsewhere Google finally released a way to get book availability on Google Books. It's a fairly simple web service where you hand it an identifier and it gives back whether the book is on Google Books and how much so (no view, partial view, full view) along with cover images. AADL previously scraped for the information which I've now disabled and used the new API instead. It was fairly easy with a little jquery and small changes to catwrap. I'm not yet using ThingISBN/xISBN so the results aren't that great. LCCN/OCLC identifiers would probably also help.

And as Tim at LibraryThing and jrochkind found they seem to be going for basic and client side which limits what you can do with it. I haven't had time to look into what's allowed in terms of caching or how many requests per client before being shut down. It would probably help to do the calls server side and cache for all the lccn/oclc/isbn possibilities in the long run.

Vanishing Librarians

2008-02-21T00:00:00+00:00

From Library Journal: Blatant Berry: The Vanishing Librarians

Our catalogers began to disappear with the takeover of that function by OCLC, the nonprofit that aspires to be a corporation in this brave new retail library world. The standardized result of the effort is bypassed by patron and librarian alike, as they turn to the more friendly Amazons, Googles, et al., for the less precise, more watered-down "metadata" that has replaced what used to be cataloging. Apparently, users don't miss the old catalog, except as a familiar artifact, which is testimony to how low this dumbing down has taken us.

...

The resulting "destination" libraries resemble the cookie-cutter design of the grocery store, aimed at making sure everyone who comes in goes out with "product" (books, CDs, DVDs, or downloads). What the patron takes is of as little concern to the storekeeper librarian as it is to the supermarket manager. The success of the enterprise is measured in the number of products collected by patrons, now called "customers." It is no longer measured in the usefulness or impact of the service on the quality of life in the community served.