What’s in Fluidinfo? An Analysis of About Tags¶

The First Approximation¶

Over the last six months or so I’ve built up a way of classifying about tags in Fluidinfo to show. This is what that (very crude) analysis shows.

Points to note particularly:

• There are 22,381 about tags (c. 1.74%) that I haven’t classified (of which many are books and authors; see blow)
• There are many (some hundreds of thousands) of objects in Fluidinfo with no about tag; those are excluded from this analysis
• The analysis is all based on pattern matching on the about tag and as such is approximate and makes assumptions.
• This analysis is only concerned with objects in Fluidinfo that have about tags; most of the data in Fluidinfo is stored as tags and tag values within the system, and some people prefer anonymous objects.

We can view this visually too. Although pie charts are generally not my favourite way of showing data, in this case I think it works quite well.

It’s interesting also to look at just the objects that have been added in the last 3 months or so (about 350,000 objects). These are as follows:

Exploring the Others¶

Although there will be a small number of misclassificaions in the figures above, clearly the goal of any exercise such as this is to classify as much as possible correctly, and while 1.74% unclassified doesn’t sound a lot it’s still over 20,000 objects. The thing that makes them hard, at least using the methodology of analysing about tags, as that (as far as I can see) there’s little in the structure of the remaining about tags to allow them to be classified.

Looking at them by eye, however, it’s immediately obvious that a lot of them (perhaps even most of them) are concerned with book in one form or another.

A lot of them are clearly book titles. Fluidinfo user librivox.org has tagged 4301 books with (among others) a librivox.org/book/title. An example is the object with about tag how to cook fish.

fdb count -q 'has librivox.org/book/title'
4301 objects matched
Total: 4301 objects

If we have a look at the tags on this object, we see this:

[10]> fdb tags -a 'how to cook fish'
Object with about="how to cook fish" (id 16e181f4-1966-4307-9a9b-e06679d56e76):
  /fluiddb/about="how to cook fish"
  /librivox.org/book/category="Non-fiction"
  /librivox.org/book/author="Green, Olive"
  /librivox.org/languages/english
  /librivox.org/genres/cookery
  /librivox.org/book/completion-timestamp=1181174400.0
  /librivox.org/book/description="<p>Olive Green is the pseudonym for the prolific late 19th Century/early 20th Ce"
  /librivox.org/book/related-authors={Non-primative type}
  /librivox.org/book/author2="Reed, Myrtle"
  /librivox.org/book/id=530
  /librivox.org/book/copyrightyear=1908
  /librivox.org/book/etext="http://www.gutenberg.org/etext/18542"
  /librivox.org/book/completion-month=6
  /librivox.org/book/completed="Thu, 07 Jun 2007 00:00:00 -0700"
  /librivox.org/book/language="English"
  /librivox.org/book/title="How to Cook Fish"
  /librivox.org/book/zipfile="http://www.archive.org/download/how_cook_fish_librivox/how_cook_fish_librivox_64"
  /librivox.org/book/completion-year=2007
  /librivox.org/book/genre={Non-primative type}
  /librivox.org/genres/advice
  /librivox.org/genres/instruction
  /librivox.org/book/rssurl="http://librivox.org/bookfeeds/how-to-cook-fish-by-olive-green.xml"
  /librivox.org/book/completion-day=7

So clearly librivox.org is simply using the title of the book (in lower case) as the about tag. Librivox also has an object for each author. So, for example, we see:

[11]> fdb tags -a 'olive green'
Object with about="olive green" (id c9e66af9-e25b-4c7e-853e-d49f2a9305c7):
  /librivox.org/author/genre={Non-primative type}
  /fluiddb/about="olive green"
  /librivox.org/genres/instruction
  /librivox.org/author/related-collaborators={Non-primative type}
  /librivox.org/author/name="Green, Olive"
  /librivox.org/author/languages={Non-primative type}
  /librivox.org/author/related-titles={Non-primative type}
  /librivox.org/languages/english
  /librivox.org/genres/cookery
  /librivox.org/genres/advice
  /librivox.org/author/collaborators={Non-primative type}
  /librivox.org/author/titles={Non-primative type}

Again, we can count these:

fdb count -q 'has librivox.org/author/name'
1692 objects matched
Total: 1692 objects

So of our 22,381, that leaves 22,381 – 4,301 – 1,692 = 16,388.

There are also a set of books using an about tag convention of the form "Title//Author" (e.g. Foundation//Isaac Asimov. I’d forgotten, but I created those before coming up with the book-1 and book-u conventions. But there are only 60 of those.

More interestingly, despite the fact that oreilly.com has used the book-u conventions, there are also objects for many or all of the O’Reilly books that just use the plain titles. Let’s have a look at one:

fdb tags -a "head first python"
Object with about="head first python" (id 5b726d3a-4a65-4ea2-bb0a-eb17a0883672):
  /fluiddb/about="head first python"
  /oreilly.com/related-objects="book:head first python (paul barry)"
  /njr/index/about

So this appears to be a kind of redirect: the key tag is oreilly.com/related-objects, which acts as a redirect to the object itself. If we count those:

fdb count -q 'has oreilly.com/related-objects'
6802 objects matched
Total: 6802 objects

we find 6,802 redirects, leaving 9,586 unclassified objects, or about 0.74%.

We could go further, but I’m going to leave it here for now. The grand total for the numer of book objects is now the 3,551 starting book:, the 4,301 from librivox.org and the 60 // books, giving a total of 7,912. Similarly, for authors, there are the 2,069 tagged author: plus the 1,692 “naked” authors from librivox.org giving a total of 3,761.

I’ll leave with a summary of my best assessment of what’s in Fluidinfo right now, taking into account the extra digging described above.