On fooling around with triples

I eventually want to be able to do this kind of analysis on "the big one", LCSH. But that really is big and queries against it will take … a while … so I've chosen to start with something smaller, the Library of Congress Demographic Group Terms. I chose this in particular because it's a reasonable size but also contains terms that appear on a list of those government departments are being instructed to remove so seems likely to be a target of that censorship.

This data is all in the form of RDF triples. This post isn't the place for a full explanation of what that means, but in brief, a triple is a machine-readable statement of some property of a subject in the form:

<subject> <predicate> <object>

So I have two versions of the same RDF ontology, and I want to know what statements have been removed or altered from one to the other. I could process the raw files myself as text but: 1) I would eventually end up writing the bits of a triple-parser and store I need myself, from scratch, which would be a waste of time; and 2) I actually would like to learn more about the key technologies involved.

To work with RDF data, I needed a specialised database called a triplestore, which is optimised for running queries against this type of graph data using a query language whimsically named SPARQL. I didn't give this too much thought, and picked Apache Jena Fuseki because it was open source, was available as a package in my Linux distribution and worked first time when I tried starting it. Other triplestores are available.

The basic unit of a triplestore is a "graph", equivalent to a "database" in a database management system. Because these are two versions of the same data I can't just load them both up into the same graph because they have significant overlap and the second would largely replace the first leaving me no way to compare them. What I can do is load them up into separate "named graphs", which I can then refer to explicitly in my queries. I've called the graphs for the two versions urn:demographicTerms/20250314 and urn:demographicTerms/20250321, for the dates I downloaded them.

Since RDF triples tend to use URIs (technically IRIs) all over the place, which tend to be quite verbose, SPARQL allows you to define prefixes to make queries less verbose. Here are a few that might be useful:

  PREFIX owl: <http://www.w3.org/2002/07/owl#>
  PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
  PREFIX mads: <http://www.loc.gov/mads/rdf/v1#>
  PREFIX ri: <http://id.loc.gov/ontologies/RecordInfo#>

I'll also define my own prefix, dt: to make it easier to refer to the two named graphs:

  PREFIX dt: <urn:demographicTerms/>

You'll see these abbreviated to <<common-prefixes>> in the code below. With that set up, let's dive into the data.

To make sure that I'm successfully querying the right place, and get an idea of what data is in there, let's pull out a list of all the unique predicates: the things that are used to make statements about other things.

  <<common-prefixes>>

  SELECT DISTINCT ?predicate
  FROM dt:20250314
  WHERE {
    [] ?predicate []
  }
  ORDER BY ?predicate

predicate
http://id.loc.gov/ontologies/RecordInfo#languageOfCataloging
http://id.loc.gov/ontologies/RecordInfo#recordChangeDate
http://id.loc.gov/ontologies/RecordInfo#recordContentSource
http://id.loc.gov/ontologies/RecordInfo#recordStatus
http://id.loc.gov/ontologies/bflc/marcKey
http://id.loc.gov/vocabulary/identifiers/lccn
http://www.loc.gov/mads/rdf/v1#adminMetadata
http://www.loc.gov/mads/rdf/v1#authoritativeLabel
http://www.loc.gov/mads/rdf/v1#citationNote
http://www.loc.gov/mads/rdf/v1#citationSource
http://www.loc.gov/mads/rdf/v1#citationStatus
http://www.loc.gov/mads/rdf/v1#deletionNote
http://www.loc.gov/mads/rdf/v1#elementList
http://www.loc.gov/mads/rdf/v1#elementValue
http://www.loc.gov/mads/rdf/v1#exampleNote
http://www.loc.gov/mads/rdf/v1#hasBroaderAuthority
http://www.loc.gov/mads/rdf/v1#hasEarlierEstablishedForm
http://www.loc.gov/mads/rdf/v1#hasMADSCollectionMember
http://www.loc.gov/mads/rdf/v1#hasNarrowerAuthority
http://www.loc.gov/mads/rdf/v1#hasReciprocalAuthority
http://www.loc.gov/mads/rdf/v1#hasSource
http://www.loc.gov/mads/rdf/v1#hasVariant
http://www.loc.gov/mads/rdf/v1#historyNote
http://www.loc.gov/mads/rdf/v1#isMemberOfMADSCollection
http://www.loc.gov/mads/rdf/v1#isMemberOfMADSScheme
http://www.loc.gov/mads/rdf/v1#note
http://www.loc.gov/mads/rdf/v1#variantLabel
http://www.w3.org/1999/02/22-rdf-syntax-ns#first
http://www.w3.org/1999/02/22-rdf-syntax-ns#rest
http://www.w3.org/1999/02/22-rdf-syntax-ns#type
http://www.w3.org/2000/01/rdf-schema#comment
http://www.w3.org/2000/01/rdf-schema#label

This gives me a little confidence that we are at least looking at the right sort of data, and also gives hints about the structure of the dataset. Now we can start digging in a bit more deeply.

In common with a lot of such vocabularies, those produced by LoC are generally quite scrupulous about retaining deleted entries but clearly marking them as such, rather than simply dropping them from the dataset. This is important: if you make use of a term that is subsequently deprecated, you need to know when that happened and why to understand how to appropriately update your own records, and historical records will likely still reference terms that have since been deleted but still need to be interpreted.

Let's take a look at what's been officially deprecated from this vocabulary:

  <<common-prefixes>>

  SELECT ?label ?delDate ?note FROM dt:20250321
  WHERE {
    OPTIONAL { ?x mads:variantLabel ?label } .
    ?x rdf:type mads:Authority .
    ?x mads:deletionNote ?delNote .
    ?x mads:adminMetadata ?md .
    ?md ri:recordChangeDate ?delDate .
    ?md ri:recordStatus "deprecated" .
    BIND(REPLACE(?delNote, "\n", " ") as ?note)
  }
  ORDER BY ?delDate

There's actually nothing problematic here, as far as I can tell. Yes, some terms have been removed over time but this is because they are covered by some other combination (e.g. Parents of dyslexics), weren't correctly constructed to start with (e.g. United States Coast Guard officers), or were duplicates (e.g. Junior high school students).

Let's move on…

There are several subsets defined within the full set of terms. One in particular, http://id.loc.gov/authorities/demographicTerms/collection_LCDGT_Social, contains terms referring to social groups, which includes various categories which are actively threatened by the current regime. These seem worth looking over to see if there's anything amiss.

  <<common-prefixes>>

  SELECT * FROM dt:20250321
  WHERE {
    ?term mads:authoritativeLabel ?label .
    ?term mads:isMemberOfMADSCollection <http://id.loc.gov/authorities/demographicTerms/collection_LCDGT_Social>
  }
  LIMIT 20

I've limited the number of results here to 20, but I've scanned through the full collection of 325 and all the terms I'd expect to see removed (relating to sexuality & gender identity, for example) are still present. For example:

  <<common-prefixes>>

  SELECT * FROM dt:20250321
  WHERE {
    ?term mads:authoritativeLabel ?label .
    FILTER(CONTAINS(LCASE(?label), "gender"))
  }

The check I've been building up to, though, is to directly compare snapshots of the vocabulary taken one week apart. This is why I uploaded two versions into named graphs, and requires slightly more verbose queries to compare.

I thought about a few ways of doing this, but I've landed on this simple option which should nonetheless identify the most obvious kinds of vandalism: what (textual) labels are present in the earlier snapshot but absent in the later? This should catch both deleted and modified terms, which we can then potentially inspect further.

In SPARQL we can do this by querying the earlier graph for all its term labels, then filtering out any of these that do not exist in the later graph:

  <<common-prefixes>>

  SELECT ?term ?label
  FROM NAMED dt:20250314
  FROM NAMED dt:20250321
  WHERE {
    GRAPH dt:20250314 {
      ?term mads:authoritativeLabel ?label
    }
    FILTER NOT EXISTS {
      GRAPH dt:20250321 { ?x mads:authoritativeLabel ?label }
    }
  }
  LIMIT 200

In this case, this returns only one result:

Running the same query reversed (what is present in the later but not the earlier graph?) also shows only one result, and it's for the same ID.

  <<common-prefixes>>

  SELECT ?term ?label
  FROM NAMED dt:20250314
  FROM NAMED dt:20250321
  WHERE {
    GRAPH dt:20250321 {
      ?term mads:authoritativeLabel ?label
    }
    FILTER NOT EXISTS {
      GRAPH dt:20250314 { ?x mads:authoritativeLabel ?label }
    }
  }
  LIMIT 200

This differs only in the case of the single letter "A", so we can reasonably assume that this is just a typo being corrected or something of that nature.

So far, it looks like this particular vocabulary has not yet been defaced. That doesn't mean it won't be though, so I'll need to continue taking snapshots at regular intervals and repeat these tests to see if anything else changes. One follow-up task, then, is to better automate this so I can't forget to do it (I've already missed the 2025-03-28 update).

There are other vocabularies that we do know have already been changed. Library of Congress Subject Headings (LCSH) are used in English-language library catalogues around the world, so the recent widely-discussed changes of the terms for "Mexico, Gulf of" and "Mount Denali" to "America, Gulf of" and "Mount McKinley" respectively will have global impact.

I'm aware that these changes do not go unnoticed in the GLAM community, since many catologuing professionals will routinely check updates to LCSH and friends before applying them to their own catalogue. I'm still doing this for my own interest, and also because a growing number of organisations simply don't have the capacity to do such checks.

LCSH is a much bigger dataset, and I plan to dig into that for my next post on this subject.