eRambler

Data rescue for World Digital Preservation Day 2025

Thu, 06 Nov 2025 21:09:58 +0000

Today, Thursday 6 November 2025 if I actually manage to finish and publish this today, is World Digital Preservation Day so I thought I would try and get a blog post out about some work I’ve been doing to rescue at-risk data. I’ve briefly mentioned this in my post about Library of Congress Subject Headings but not in much detail.

The project is Safeguarding Research & Culture and I got involved back in March or April when Henrik reached out on social media looking for someone with library & metadata experience to contribute. I said that I wasn’t a Real Librarian but I’d love to help if I could, and now here we are.

The concept is simple: download public datasets that are at risk of being lost, and replicate them as widely as possible to make them hard to destroy, though obviously there’s a lot of complexity buried in that statement. When the Trump administration first took power, there were a lot of people around the world worried about this issue and wanting to help, so while there are a number of institutions & better resourced groups doing similar things, we aim to complement them by mobilising grassroots volunteers.

Downloading data isn’t always straightforward. It may be necessary to crawl an entire website, or query a poorly-documented API, or work within the constraints of rate-limiting so as not to overload an under-resourced server. That takes knowledge and skill, so part of the work is guiding and mentoring new contributors and fostering a community that can share what they learn and proactively find and try out new tools.

We also need people to be able to find and access the data, and volunteers to be able to contribute their storage to the network. We distribute data via the venerable BitTorrent protocol, which is very good at defeating censorship and getting data out to as many peers as possible as quickly as possible. To make those torrents discoverable, our dev team led by the incredible Jonny have built a catalogue of dataset torrents, playfully named SciOp. That’s built on well-established linked data standards like DCAT, the Data Catalogue Vocabulary, so the metadata is standardised and interoperable, and there’s a public API and a developing commandline client to make it even easier to process and upload datasets. There are even RSS and RDF feeds of datasets by tag, size, threat status or number of seeds (copies) in the network that you can plug into your favourite BitTorrent client to automatically start downloading newly published datasets. There are even exciting plans in the works to make it federated via ActivityPub, to give us a network of catalogues instead of just a single one.

We’re accidentally finding ourselves needing to push the state of the art in BitTorrent client implementations. If you’re familiar with the history of BitTorrent as a favoured tool for ahem less-than-legal media sharing, it probably won’t surprise you that most current BitTorrent clients are optimised for working with single audio-visual streams of about 1 to 2½ hours in length. Our scientific & cultural data is much more diverse than that, and the most popular clients can struggle for various reasons. In many cases there are BEPs (BitTorrent Enhancement Proposals) to extend the protocol to improve things, but these are optimal features that most clients don’t implement. The collection of BEPs that make up “BitTorrent v2” is a good example: most clients don’t support v2 well, so most people don’t bother making v2-compatible torrents, but that means there’s no demand to implement v2 in the clients. We are planning to make a scientific-grade BitTorrent client as a test-bed for these and other new ideas.

Myself I’m running one of a small number of “super” nodes in the swarm, with much more storage available than the average laptop or desktop, and often much better bandwidth too. That’s good, because some of our datasets run to multiple terabytes, plus to ensure new nodes can get started quickly we need to have some always-on nodes with most of the data available to others. Since BitTorrent is truly peer-to-peer, it doesn’t matter how many people have a copy of a given dataset, if none of them are online no-one else can access it.

This is all very technically interesting, but communications, community, governance, policy, documentation, funding are also vitally important, and for us these are all works in progress. We need volunteers to help with all of this, but especially those less-technical aspects. If you’re interested in helping, please drop us a line at contact@safeguar.de, or join our community forum and introduce yourself and your interests.

If you want to contribute but don’t feel you have the time or skills, well, to start with we’re more than happy to show you the ropes and help you get started, but as an alternative, I’m running one of those “super” nodes and you can contribute to my storage costs via GoFundMe: even a few quid helps. I currently have 3x 6TB hard drives with no space to mount them, so I’m currently in need of a drive cage to hold them and plug them into my server.

Special shout-out also to our sibling project, the Data Rescue Project, who are doing amazing work on this and often send us requests for websites or complex datasets for our community to save.

I’ve barely scratched the surface here, but I really want to actually get this post out for WDPD so I’m going to stop here and hopefully continue soon!

Trying to rediscover my voice

Mon, 21 Jul 2025 21:13:58 +0100

I always feel like I have so much to write, so much that I want to write. So why is it that anytime I sit down to actually write my mind goes entirely blank? Even if I have a list of intended topics right in front of me, or a partial draft to work on, I start feeling like I have nothing interesting to say. The only thing I have much success in writing is these long, boring, self indulgent walls of text about how I feel about not being able to write. I don’t know, maybe I should just publish this. At least it will be some of my thoughts out in the world again.

Back in the early days of my blog it seemed to flow quite easily. I was confident, not really that what I had to say was correct, or insightful, or in any way important, but that it was not going to get me into trouble or draw criticism that I couldn’t handle. I naively thought that I could say whatever I wanted without fear.

Since I am a white, well-educated, straight cis man, I was largely right. The only time I felt any discomfort was when I posted a paraphrased recipe from a book and was threatened with legal action by the writer’s agent. While I was certain that legally I was in the clear (you can’t copyright a recipe), and felt that since I was strongly recommending people buy the book it was OK morally too, I deleted the post.

Of course I did. I’ve spent my whole life learning over and over that conflict is bad, it’s my responsibility to resolve it, and it probably is me being unreasonable anyway. Like a lot of ND kids, I learned early on that all these intense feelings and sense impressions (that I believed everyone felt) were not to be acted upon because doing so only brought trouble.

So I think after that experience, benign though it was, I started to doubt whether I could speak freely after all. At the same time I was awakening politically and learning that some of the beliefs I thought were obvious (e.g. that all humans had rich inner lives that affected how they thought/acted) were in fact not that common, and that stating them in particular ways could be seen as somehow controversial. I was also succeeding in my career and starting to internalise what I was told about what I said reflecting on my employer and colleagues.

However it happened, I had lost my voice.

Yes, in the sense of being literally unable to express myself in certain ways, but also in that I felt there was something I once had but had misplaced and desperately wanted to find again without knowing where to look. First it only affected my personal writing, but as the years went by it crept into my professional work too. Working in a large organisation is never not political; what you say affects not only your own standing and influence but also that of your team and department in the wider organisation, which is a heck of a responsibility. By the time I ended up in the public sector I was being regularly reminded that it was my job to remain neutral, impartial, disinterested. Eventually I stopped saying much at all, except to a few trusted friends and colleagues.

This is not my fault, exactly, but it is partly related to some very core parts of my personality. I know that breaking rules feels bad, feels dangerous. So when I’m told implicitly that regardless of the depth of my knowledge or experience my word isn’t good enough —that only things supported by concrete evidence are OK to say— I play it safe.

I see that not everyone plays by those rules, and that some don’t suffer any consequences for breaking them, but I’m unable to discern to my own satisfaction why that is or how to emulate it. Is it because they are brighter or more experienced than I? Are they party to information that explains how they are not, in fact, breaking rules? Do they have better understanding of what rules can or cannot be broken? Are they more mature and self-confident through experience? Are they simply confident because they have the privilege never to have been challenged and the power to carry it off through confidence alone? Maybe some of these are people I don’t want to emulate after all…

Still, there are glimmers of hope. I’m starting to become more aware of contexts where I feel less shackled and more able to express myself. Unsurprisingly, it’s usually when I’m under less pressure (external or internal) to “deliver” some “output” that meets some vague criteria, and when I’m working with people I know well and trust not to judge me personally if we disagree. It’s also easier when I retreat to that shrinking zone where I still feel like I can speak with some authority.

I’m looking for ways to put myself in that context more often. Right now I think that means identifying a small group of trusted colleagues at work that I can bounce ideas around with, and doing more writing in the various communities I find myself in. There was a long while when I didn’t feel I had the authority to speak even about my own lived experience. Two years of therapy, a lot of introspection, and the love of friends and family have brought me to a place where I no longer doubt my own experience of the world (well, not so much as I did — it’s a work in progress), which gives me a place of solid ground to build out from as I re-establish my faith in my skills, experience and judgement in other areas.

Well, this wasn’t the thing I was expecting to write when I started, but here we are. I guess we’ll see how it goes!

On fooling around with triples

Sun, 06 Apr 2025 00:00:00 +0000

Content Note

For reasons that will become apparent as you read the introduction, this post has tangential references to a number of oppressed groups and other fascist newspeak as encoded in Library of Congress metadata standards.

If you've been aware at all about the current political situation in the US, you'll be aware that the dude in the Oval Office has been issuing executive orders left, right and centre. Much damage is being done to the country, its people and its infrastructure that will take generations to repair, but as someone watching from an ocean away it would be easy to believe it won't affect me.

It will.

Not only will any success experienced by fascism in the US embolden fascists around the world, but the US is so culturally and economically dominant that anything happening there will inevitably impact the rest of us. "When America sneezes, the rest of the world catches a cold."

If this all seems like a weird way to introduce a post about me figuring out how to query and manipulate RDF metadata, well, it is. I do intend to write more about this soon, but for now I'll just say that the question I'm interested in is this:

Is the linked data published by the Library of Congress being materially changed as a result of the aforementioned flurry of executive orders?

If you think this sounds hypothetical, well, the US National Cancer Institute has already begun removing certain gender- and sexuality-related terms from its linked data thesaurus. I like to believe that, for reasons of both professional integrity and institutional inertia, it will take much longer before similar changes are made to vocabularies like the Library of Congress Subject Headings (LCSH). These are resources used in cultural heritage institutions all over the world, though, so if & when that happens the impact will be widespread.

The setup

I eventually want to be able to do this kind of analysis on "the big one", LCSH. But that really is big and queries against it will take … a while … so I've chosen to start with something smaller, the Library of Congress Demographic Group Terms. I chose this in particular because it's a reasonable size but also contains terms that appear on a list of those government departments are being instructed to remove so seems likely to be a target of that censorship.

This data is all in the form of RDF triples. This post isn't the place for a full explanation of what that means, but in brief, a triple is a machine-readable statement of some property of a subject in the form:

<subject> <predicate> <object>

So I have two versions of the same RDF ontology, and I want to know what statements have been removed or altered from one to the other. I could process the raw files myself as text but: 1) I would eventually end up writing the bits of a triple-parser and store I need myself, from scratch, which would be a waste of time; and 2) I actually would like to learn more about the key technologies involved.

To work with RDF data, I needed a specialised database called a triplestore, which is optimised for running queries against this type of graph data using a query language whimsically named SPARQL. I didn't give this too much thought, and picked Apache Jena Fuseki because it was open source, was available as a package in my Linux distribution and worked first time when I tried starting it. Other triplestores are available.

The basic unit of a triplestore is a "graph", equivalent to a "database" in a database management system. Because these are two versions of the same data I can't just load them both up into the same graph because they have significant overlap and the second would largely replace the first leaving me no way to compare them. What I can do is load them up into separate "named graphs", which I can then refer to explicitly in my queries. I've called the graphs for the two versions urn:demographicTerms/20250314 and urn:demographicTerms/20250321, for the dates I downloaded them.

Since RDF triples tend to use URIs (technically IRIs) all over the place, which tend to be quite verbose, SPARQL allows you to define prefixes to make queries less verbose. Here are a few that might be useful:

  PREFIX owl: <http://www.w3.org/2002/07/owl#>
  PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
  PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
  PREFIX mads: <http://www.loc.gov/mads/rdf/v1#>
  PREFIX ri: <http://id.loc.gov/ontologies/RecordInfo#>

I'll also define my own prefix, dt: to make it easier to refer to the two named graphs:

  PREFIX dt: <urn:demographicTerms/>

You'll see these abbreviated to <<common-prefixes>> in the code below. With that set up, let's dive into the data.

Getting situated

To make sure that I'm successfully querying the right place, and get an idea of what data is in there, let's pull out a list of all the unique predicates: the things that are used to make statements about other things.

  <<common-prefixes>>

  SELECT DISTINCT ?predicate
  FROM dt:20250314
  WHERE {
    [] ?predicate []
  }
  ORDER BY ?predicate

predicate
http://id.loc.gov/ontologies/RecordInfo#languageOfCataloging
http://id.loc.gov/ontologies/RecordInfo#recordChangeDate
http://id.loc.gov/ontologies/RecordInfo#recordContentSource
http://id.loc.gov/ontologies/RecordInfo#recordStatus
http://id.loc.gov/ontologies/bflc/marcKey
http://id.loc.gov/vocabulary/identifiers/lccn
http://www.loc.gov/mads/rdf/v1#adminMetadata
http://www.loc.gov/mads/rdf/v1#authoritativeLabel
http://www.loc.gov/mads/rdf/v1#citationNote
http://www.loc.gov/mads/rdf/v1#citationSource
http://www.loc.gov/mads/rdf/v1#citationStatus
http://www.loc.gov/mads/rdf/v1#deletionNote
http://www.loc.gov/mads/rdf/v1#elementList
http://www.loc.gov/mads/rdf/v1#elementValue
http://www.loc.gov/mads/rdf/v1#exampleNote
http://www.loc.gov/mads/rdf/v1#hasBroaderAuthority
http://www.loc.gov/mads/rdf/v1#hasEarlierEstablishedForm
http://www.loc.gov/mads/rdf/v1#hasMADSCollectionMember
http://www.loc.gov/mads/rdf/v1#hasNarrowerAuthority
http://www.loc.gov/mads/rdf/v1#hasReciprocalAuthority
http://www.loc.gov/mads/rdf/v1#hasSource
http://www.loc.gov/mads/rdf/v1#hasVariant
http://www.loc.gov/mads/rdf/v1#historyNote
http://www.loc.gov/mads/rdf/v1#isMemberOfMADSCollection
http://www.loc.gov/mads/rdf/v1#isMemberOfMADSScheme
http://www.loc.gov/mads/rdf/v1#note
http://www.loc.gov/mads/rdf/v1#variantLabel
http://www.w3.org/1999/02/22-rdf-syntax-ns#first
http://www.w3.org/1999/02/22-rdf-syntax-ns#rest
http://www.w3.org/1999/02/22-rdf-syntax-ns#type
http://www.w3.org/2000/01/rdf-schema#comment
http://www.w3.org/2000/01/rdf-schema#label

This gives me a little confidence that we are at least looking at the right sort of data, and also gives hints about the structure of the dataset. Now we can start digging in a bit more deeply.

Deprecated records

In common with a lot of such vocabularies, those produced by LoC are generally quite scrupulous about retaining deleted entries but clearly marking them as such, rather than simply dropping them from the dataset. This is important: if you make use of a term that is subsequently deprecated, you need to know when that happened and why to understand how to appropriately update your own records, and historical records will likely still reference terms that have since been deleted but still need to be interpreted.

Let's take a look at what's been officially deprecated from this vocabulary:

  <<common-prefixes>>

  SELECT ?label ?delDate ?note FROM dt:20250321
  WHERE {
    OPTIONAL { ?x mads:variantLabel ?label } .
    ?x rdf:type mads:Authority .
    ?x mads:deletionNote ?delNote .
    ?x mads:adminMetadata ?md .
    ?md ri:recordChangeDate ?delDate .
    ?md ri:recordStatus "deprecated" .
    BIND(REPLACE(?delNote, "\n", " ") as ?note)
  }
  ORDER BY ?delDate

label	delDate	note
Prisoners of war	2021-01-22T15:39:00	This authority record was deleted because it was created in error.
Concentration camp inmates	2022-11-09T12:41:19	This authority record has been deleted because the demographic group term is covered by the demographic group terms {Internment camp inmates} (DLC)dg2022060247 and {Nazi concentration camp inmates} (DLC)dg2022060248
Parents of autistics	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of autistics} (DLC)dg2024060011 and {Parents} (DLC)dg2015060230
Parents of mass murderers	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of mass murderers} (DLC)dg2024060075 and {Parents} (DLC)dg2015060230
Politicians' partners	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of politicians} (DLC)dg2024060009 {Spouses} (DLC)dg2024060004 and {Unmarried partners} (DLC)dg2024060005
Cancer patients' partners	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of cancer patients} (DLC)dg2024060007 {Spouses} (DLC)dg2024060004 and {Unmarried partners} (DLC)dg2024060005
Parkinson's disease patients' partners	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of Parkinson's disease patients} (DLC)dg2024060008 {Spouses} (DLC)dg2024060004 and {Unmarried partners} (DLC)dg2024060005
Parents of transgender people	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of transgender people} (DLC)dg2024060076 and {Parents} (DLC)dg2015060230
Parents of dyslexics	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of dyslexics} (DLC)dg2024060073 and {Parents} (DLC)dg2015060230
Partners (Spouses)	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Spouses} (DLC)dg2024060004 and {Unmarried partners} (DLC)dg2024060005
Parents of alcoholics	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of alcoholics} (DLC)dg2024060010 and {Parents} (DLC)dg2015060230
Parents of gays	2024-06-06T12:09:08	This authority record has been deleted because the heading is covered by the headings {Family members of gay people} (DLC)dg2024060074 and {Parents} (DLC)dg2015060230
United States Air Force officers	2024-08-21T13:21:46	This authority record has been deleted because it is not a valid construction.
United States Coast Guard officers	2024-08-21T13:21:46	This authority record has been deleted because it is not a valid construction.
Junior high school students	2025-01-24T13:29:48	This authority record has been deleted because the heading is covered by the heading {Middle school students} (DLC)dg2015060024

There's actually nothing problematic here, as far as I can tell. Yes, some terms have been removed over time but this is because they are covered by some other combination (e.g. Parents of dyslexics), weren't correctly constructed to start with (e.g. United States Coast Guard officers), or were duplicates (e.g. Junior high school students).

Let's move on…

Social terms (includes gender and sexuality)

There are several subsets defined within the full set of terms. One in particular, http://id.loc.gov/authorities/demographicTerms/collection_LCDGT_Social, contains terms referring to social groups, which includes various categories which are actively threatened by the current regime. These seem worth looking over to see if there's anything amiss.

  <<common-prefixes>>

  SELECT * FROM dt:20250321
  WHERE {
    ?term mads:authoritativeLabel ?label .
    ?term mads:isMemberOfMADSCollection <http://id.loc.gov/authorities/demographicTerms/collection_LCDGT_Social>
  }
  LIMIT 20

label	term
Labor union members	http://id.loc.gov/authorities/demographicTerms/dg2023060130
Family members of Alzheimer's patients	http://id.loc.gov/authorities/demographicTerms/dg2015060782
Clavichord students	http://id.loc.gov/authorities/demographicTerms/dg2016060518
Males	http://id.loc.gov/authorities/demographicTerms/dg2015060003
Iraq War veterans	http://id.loc.gov/authorities/demographicTerms/dg2024060120
Chemistry students	http://id.loc.gov/authorities/demographicTerms/dg2017060075
Greek, Students of	http://id.loc.gov/authorities/demographicTerms/dg2023060042
Family members of mass murderers	http://id.loc.gov/authorities/demographicTerms/dg2024060075
Quechua, Students of	http://id.loc.gov/authorities/demographicTerms/dg2023060041
Orphans	http://id.loc.gov/authorities/demographicTerms/dg2022060429
Czech, Students of	http://id.loc.gov/authorities/demographicTerms/dg2022060118
Navajo, Students of	http://id.loc.gov/authorities/demographicTerms/dg2023060063
Children of politicians	http://id.loc.gov/authorities/demographicTerms/dg2017060126
Expatriates	http://id.loc.gov/authorities/demographicTerms/dg2015060864
Holocaust victims	http://id.loc.gov/authorities/demographicTerms/dg2015060152
Two-spirit people	http://id.loc.gov/authorities/demographicTerms/dg2022060056
Booker Prize winners	http://id.loc.gov/authorities/demographicTerms/dg2022060316
Prix Carbet de la Caraïbe et du Tout-Monde winners	http://id.loc.gov/authorities/demographicTerms/dg2024060112
Stepbrothers	http://id.loc.gov/authorities/demographicTerms/dg2015060238
Parents	http://id.loc.gov/authorities/demographicTerms/dg2015060230

I've limited the number of results here to 20, but I've scanned through the full collection of 325 and all the terms I'd expect to see removed (relating to sexuality & gender identity, for example) are still present. For example:

  <<common-prefixes>>

  SELECT * FROM dt:20250321
  WHERE {
    ?term mads:authoritativeLabel ?label .
    FILTER(CONTAINS(LCASE(?label), "gender"))
  }

label	term
Family members of transgender people	http://id.loc.gov/authorities/demographicTerms/dg2024060076
Gender minorities	http://id.loc.gov/authorities/demographicTerms/dg2015060398
Transgender people	http://id.loc.gov/authorities/demographicTerms/dg2015060006
Cisgender people	http://id.loc.gov/authorities/demographicTerms/dg2017060283
Gender studies teachers	http://id.loc.gov/authorities/demographicTerms/dg2017060049
Genderqueer people	http://id.loc.gov/authorities/demographicTerms/dg2022060046

What's changed?

The check I've been building up to, though, is to directly compare snapshots of the vocabulary taken one week apart. This is why I uploaded two versions into named graphs, and requires slightly more verbose queries to compare.

I thought about a few ways of doing this, but I've landed on this simple option which should nonetheless identify the most obvious kinds of vandalism: what (textual) labels are present in the earlier snapshot but absent in the later? This should catch both deleted and modified terms, which we can then potentially inspect further.

In SPARQL we can do this by querying the earlier graph for all its term labels, then filtering out any of these that do not exist in the later graph:

  <<common-prefixes>>

  SELECT ?term ?label
  FROM NAMED dt:20250314
  FROM NAMED dt:20250321
  WHERE {
    GRAPH dt:20250314 {
      ?term mads:authoritativeLabel ?label
    }
    FILTER NOT EXISTS {
      GRAPH dt:20250321 { ?x mads:authoritativeLabel ?label }
    }
  }
  LIMIT 200

In this case, this returns only one result:

term	label
http://id.loc.gov/authorities/demographicTerms/dg2022060384	Porto-Alegrenses

Running the same query reversed (what is present in the later but not the earlier graph?) also shows only one result, and it's for the same ID.

  <<common-prefixes>>

  SELECT ?term ?label
  FROM NAMED dt:20250314
  FROM NAMED dt:20250321
  WHERE {
    GRAPH dt:20250321 {
      ?term mads:authoritativeLabel ?label
    }
    FILTER NOT EXISTS {
      GRAPH dt:20250314 { ?x mads:authoritativeLabel ?label }
    }
  }
  LIMIT 200

term	label
http://id.loc.gov/authorities/demographicTerms/dg2022060384	Porto-alegrenses

This differs only in the case of the single letter "A", so we can reasonably assume that this is just a typo being corrected or something of that nature.

What's next?

So far, it looks like this particular vocabulary has not yet been defaced. That doesn't mean it won't be though, so I'll need to continue taking snapshots at regular intervals and repeat these tests to see if anything else changes. One follow-up task, then, is to better automate this so I can't forget to do it (I've already missed the 2025-03-28 update).

There are other vocabularies that we do know have already been changed. Library of Congress Subject Headings (LCSH) are used in English-language library catalogues around the world, so the recent widely-discussed changes of the terms for "Mexico, Gulf of" and "Mount Denali" to "America, Gulf of" and "Mount McKinley" respectively will have global impact.

I'm aware that these changes do not go unnoticed in the GLAM community, since many catologuing professionals will routinely check updates to LCSH and friends before applying them to their own catalogue. I'm still doing this for my own interest, and also because a growing number of organisations simply don't have the capacity to do such checks.

LCSH is a much bigger dataset, and I plan to dig into that for my next post on this subject.

On metrics and power structures

Thu, 19 Sep 2024 00:00:00 +0000

Recently I was at the 2024 Make Data Count Summit, at Wellcome Trust HQ in London. It's a follow-on from a project of the same name which has now become a more sustained long-term effort with a growing community, focused on metrics to measure the impact of data sharing and reuse.

There was a lot for me to absorb, process and reflect on, but one thing I noticed was that some examples gave me a really icky feeling while others had me mentally cheering them on, and I've been reflecting a little on why that is. My first analysis is that it boils down to this: is this metric being used to exert control over others, or otherwise perpetuate unjust power structures?

So for example, a few of the panellists were senior academic leaders with hiring & promotion responsibility who very matter-of-factly said that, while they understood the higher goals, the fact was that they had a limited pot of money and could only fund either A or B but not both, so using "objective" metrics is both more fair and more efficient. That one had the ick factor.

On the other hand, one of the panellists (representing a research funder) was very clear that they were not using any measures of data sharing & reuse to evaluate grantholders or allocate future funding, but only to evaluate whether their own policies and initiatives to encourage good data practice were working as intended. No ick for me there.

What's the difference? In the first case, we see an example of a group with significant power making use of metrics to exercise power over others. A generous interpretation is that they are themselves subject to the power of those who assign their departmental budgets, and using an "objective", "data-driven" process at least allows them to be as fair as possible. A less generous interpretation is that use of metrics saves them time at the expense of those further down the power gradient, while soothing their own conscience and forestalling any argument with a veneer of objectivity lent by the use of data.

I don't believe anyone at the event is deserving of that second, rather cynical (who, me?) interpretation. My point is that the power differential exists and, intentional or not, this way of using metrics both obscures and perpetuates that, which is undemocratic.

It also falls into the system trap of Success to the Successful, since it awards money and power on the basis of previous success. The more funding an academic receives and the higher their rank within their institution, the easier for them to ensure the next research activity meets whatever criteria are set, whether by having more capacity (their own, or that of research assistants and students) to do the necessary work or simply having influence over the criteria themselves. Reinforcing feedback loops are incredibly powerful, and the only remedies are to break the loop or introduce an opposing balancing loop.

There were some really interesting suggestions that the way to solve this is to accept that people will try to game the system and on that basis introduce systems where gaming them still results in desired behaviour. This is interesting enough to be worth trying, but I'm skeptical of its utility in practice. Trying to predict and fix all the different unintended consequences of such an intervention requires you to out-think human nature itself. On top of that, I'm increasingly uncomfortable with the framing about individuals "gaming the system": since it's the system that permits and rewards undesirable behaviour, it's inevitable that at some point that behaviour will take place.

It seems to me that it's preferable to change the system to be able to absorb a reasonable amount of disruption while still delivering the desired outcomes. Maybe that's just a different framing of the same idea, but it seems important to be looking at the system as a whole along with its goals and those of its constituent parts. Just as important, many of those constituent parts are people towards whom we should be directing some compassion.

I've wandered off-point now (quelle surprise) so let's sum up: I think quantitative metrics are an important source of information. It's important to be aware of the shortcomings of any given metric and to only use it as one signal in a broader analysis of both quantitative and qualitative evidence. Metrics are best used by the people whose policy they are measuring, as an important feedback loop in the development of that policy. I strongly believe they should never be used to control the behaviour of others where the measurer has power over the measured, although measuring up the power gradient can be a way of holding power to account.

Economist Charles Goodhart summed it up nicely with what is commonly referred to as Goodhart's Law, commonly stated as

"When a measure becomes a target, it ceases to be a good measure"¹

I actually quite like Goodhart's own phrasing, in a nerdy way: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes"

On being told to do more with less

Thu, 08 Aug 2024 20:54:48 +0100

Why do we celebrate “efficiency” so much?

Cuts to public services are vaunted as “efficiency savings”. Free markets (and by extension capitalism though they’re not technically the same thing) are touted as the most “efficient” way to distribute resources. New technologies are sold on promises of more efficient working practices.

But we rarely question who benefits from these supposed efficiencies, and who pays the cost.

One example I’ve run into time and time again will be familiar to anyone who’s worked in a large organisation: replacing experienced specialist staff (often on finance or HR, but anything seen as a “support service” is at risk) with an IT system that enables, “self-service”.

Usually this is celebrated as a resounding success, since it’s generally followed by a reduction in headcount with no cost other than the cost of the system, which is both lower and reduces your risk because you don’t to have to make redundancy payouts when you get rid of a service provider. The thing these assessments never seem to account for is that the work doesn’t disappear: it simply moves somewhere its cost isn’t so visible on a balance sheet.

Suppose my employer has introduced a Revolutionary New System™ to let me submit my own expense claims via the web. I do this a handful of times a year, so every time I have to do it I need to:

Find the expenses policy
Check it really is the latest version
Figure out what limits apply and which categories each of my expenses fall into
Remember how to put all of that into the counter-intuitive self-service system
Apply the right accounting codes
Find a working scanner and scan the receipts
Upload those and finally hit submit

If I’ve made a mistake, it’ll be sent back to me to try and figure out what I did wrong, after which I’ll have to repeat the process.

In contrast, the finance assistant replaced by that self-service system did that (and a range of other tasks) on a daily or weekly basis. They knew the policy, they knew the edge cases, they knew the mistakes to look for and how to explain to me how to fix them. In the time it took me to submit one very average expense claim, they could process several based only on the scanned receipts they received. They were a specialist and good at their job. I’m great at my job but not so great at doing theirs too. Multiply that across every employee who travels for work, or otherwise pays out of pocket for things that need to be claimed back from the employer, and for every person you’ve made redundant, you’ve reduced your organisation’s capacity to get things done by much more by pushing specialist work onto non-specialists.

The balance sheet tells a different story though: you’ve (apparently) reduced your pay bill and all that work is (apparently) still being done, so everyone wins, must be an efficiency saving. The things those making these assertions never seem to ask is this: what work is not being done for this to continue being done by fewer people?

Alternatively, maybe everyone ups their pace, picks up the extra work and manages to continue as before, eating into any spare capacity they might have had. They’re rushing through things that they would’ve checked before, so mistakes start to slip through. When the mistakes come to light, they need to be fixed, which is more work that must be done, further eating into that spare capacity. The end of that road is catastrophe for the employer and burnout for the employee, but it will have an impact long before then.

Consider this example of a corporate invoicing scam, where a fake invoice is sent to a company’s accounts department in such a way to manipulate someone into paying the scammer as though they are a real supplier. As @joepie90 notes in that thread, and like so many others, this type of scam relies on cranking up the time pressure on the recipient, giving them less time to make an accurate assessment of how they should act. Who’s more likely to get that wrong? Someone in a team of four under pressure to pick up a workload previously carried by six.

Reducing spare capacity also hits resilience. We saw this writ large with the pandemic: the recent UK Covid-19 Inquiry Report described “public services … running close to, if not beyond capacity” after years of austerity. The National Cyber Security Centre has also hinted at this, saying that our resilience has not developed “at the pace necessary to match our adversaries”.

None of the issues I’ve just described are easy to capture on a spreadsheet. So what looks like a success on paper — a consequence-free reduction in staff costs — is in fact nothing of the sort. The costs are still there, and probably greater than before, but they’ve been shuffled around in a way that makes them hard to see.

It’s like that optical illusion where a bar of chocolate is sliced up into uneven shaped pieces, which are then apparently rearranged to make a chocolate bar the same size as the original but with one square left over. It looks like you’ve made free chocolate from thin air, but the space where that chocolate used to be is long- and thin-enough that your eye assumes the pieces just haven’t been pushed all the way together.

In other words, these amazing efficiency savings are a sleight of hand meant to hide a reduction in capacity by spreading it so thinly no one notices. In the short term we’re all fooled into doing a tiny bit more work, and the person at the top gets a nice bonus for their financial prudence. But do it over and over again and people start to notice they’re working harder than ever while getting less and less done. You can’t keep that up long without burning out large portions of your workforce.

You don’t get infinite chocolate for free.

On Donella Meadows and Systems Thinking

Tue, 18 Jun 2024 19:08:39 +0100

This weekend I started reading Donella Meadows' Thinking in Systems: A Primer and I cannot overstate how profoundly glad I am to have come across Systems Thinking as a whole field of study. It pulls together so many things that have interested me over the years, and makes sense of a whole load of things I’ve observed about the world around me.

I used to tell people that my abiding interest over time was the study of systems, but stopped because most people assumed that meant computer systems. I meant complex systems in the broadest sense, and I found it terribly frustrating that people thought I would focus only on the stereotypical white male nerd interest of “computers”.

Obviously, if you know me at all, then assuming the unsaid “computer” in systems is pretty reasonable, as it’s where I spend most of my time. Computers were the first systems I found that I could have a significant level of control over. They gave me a marketable skillset and I fitted into the stereotype so my interest in them was encouraged.

But I’ve always been just as interested in computers and software for the part they play in wider systems, especially those involving people. Maybe that’s partly why I’ve never been tempted to go for higher paid work in the software industry where I would be expected to have that narrow focus and only care about my team’s sub-sub-subsystem. I guess as an undiagnosed autistic kid, developing a special interest in the interactions and relationships between people was something that enabled me to go through the motions and fit in, which undoubtedly kept me safe, though probably at the expense of my mental health.

In any case, I am where I am now and thankfully have enough time and energy to explore this. Meadows’ work resonates with me especially because of the role she played in bringing a systems perspective to environmental and societal issues. She was the main author of the 1972 report, The Limits to Growth, which predicted that a business-as-usual approach to economic and population growth on a finite planet would inevitably lead to a collapse. Largely ignored by mainstream policymakers at the time, those predictions are looking increasingly prescient.

N.B. This post is an extended version of this Mastodon thread.

On learning to code without mathematics

Mon, 03 Jun 2024 22:06:48 +0100

Why is it that so many beginner programming tutorials assume that the learner is both a) comfortable with maths; and b) motivated to learn by seeing simple arithmetic? Go look at your favourite tutorial (I’ll wait) and I’ll give you good odds it starts with some variation of “look, you can use this as a calculator!”

Seriously, there must be so many people who would be great programmers but who never get past the thought:

“Why would I want to install and configure this whole programming environment just to see that 5×7 is 35?”

A lot of people are instantly put off when they see mathematics. That’s not a personal failure, it’s a natural consequence of the (inaccurate) way maths is seen in our culture, as a binary thing where everything is questions that have only one “right” answer and a lot of wrong ones, though it’s actually a beautiful, creative and constantly evolving language. There’s a weird little duality, often instilled in us before we even reach school, whereby on the one hand not being able to “do maths” is seen as a shameful thing while on the other it’s far more acceptable to make a general statement like “oh, I don’t really do maths” than to say “oh, I don’t really do reading”.

That perceived double bind must put off loads of people who would otherwise be very creative solvers of problems involving some maths, and when we make it look like being “good at maths” is a prerequisite of being able to program a computer we put those same people off that too. In any case I believe this is all based on a false premise: maths is much more accessible than many have been led to believe, and learning to code by manipulating text or images has a lot of potential for demystifying mathematical concepts through familiarity.

Aside: there is some evidence¹ that natural language skill is a better predictor of programming aptitude than mathematical skill, though I don’t know whether that study has been replicated.

Anyway, that’s why I really like the tutorial for the Racket programming language: it focuses 100% on drawing pictures and introduces programming concepts as ways of composing simple shapes (like squares and circles) into more complex images.

Building on this idea, Christine Lemmer-Webber has a tutorial for Digital Humanities folk called “Building a snowman with Racket” which takes the learner through making a little picture of a snowman using Racket’s slideshow module.

I don’t know about you, but I think that’s pretty cool and I’d like to see more tutorials taking a similar approach.

Best practice, or ignoring the care label

Mon, 05 Feb 2024 19:37:00 +0000

I was hanging the laundry the other day, and ended up thinking about reasons why you might ignore those coded instructions on the care label of your clothing. I came up with quite a few…

The label isn’t accessible to you (e.g. you’re blind or partially sighted)
Your needs don’t match the assumptions of the manufacturer
You habitually remove them because they itch
You don’t have the necessary equipment
You don’t understand the consequences
You don’t know what the symbols mean
You can afford to buy more clothes
Your clothes don’t have them
You don’t know they’re there
You don’t have the energy
You don’t have the time

There are certainly many more that I haven’t thought of.

Anyway, it seemed like a good example of why “best practice” doesn’t work: it suggests that there is one “best” way of doing any given task and that any other way of doing that task is necessarily inferior. It seems like a minor niggle, but I prefer “good practice” because it seems more appropriate to have a collection of good (or even “good enough”) practices that you can choose from and combine depending on your context.

Writing about my work

Thu, 07 Dec 2023 15:00:55 +0000

One of the reasons I’ve been blogging less lately is that the more I’ve progressed in my career the more aware I’ve been of representing my employer as well as myself and, let’s be honest, I have enough anxiety already without adding to it myself.

So I’ve finally had The Conversation about this with my line manager and agreed that I can write publicly about stuff that touches on my job role, as long as it’s clear that it’s my own opinion and not necessarily any statement of policy. It’s my hope that this will help with my thinking and creativity both in and out of work, as I’ll be able to get more of my opinions and ideas out there for feedback/criticism.

TL;DR Anything I publish on here is my own opinion and should not be taken as the official position of basically anyone at all, not least whoever my current employer at the time might be… 😅

Training my handwriting model: an update

Tue, 29 Aug 2023 11:22:09 +0100

My handwriting!

It’s only taken me two years, but I’ve finally got around to transcribing enough pages to have a first try at training a model on my own handwriting. My workflow has been (roughly) this:

Write something out longhand with pen and paper
Scan the pages and upload to Transkribus
Run layout detection, tweak any arrors and transcribe the text on desktop (while listening to some music)

A few useful points I’ve picked up so far:

My old, inherited multifunction printer has an option to scan directly. to FTP, and Transkribus allows upload via FTP so I have set up a scan target to send the documents straight to Transkribus from where I can add them to a collection and process them
It’s important to transcribe accurately what was on the page, not what you intended to write, but my brain tries to autocorrect as it goes so I have to concentrate on suppressing that!
Where I do notice an error while I’m writing, or change my mind about what I want to say, I try to clearly strike it out on the page then apply a gap tag while transcribing as recommended in the docs.
I’ve played a little bit with inserting Markdown-style formatting characters when I want to emphasise something, but only when it won’t disrupt my flow; for links I mostly just include <link here> as a reminder.

So now I’ve got 25 pages, the minimum recommended for training, I ran a couple of training jobs, using that as ground truth (one without and one with an existing model as basis) and the initial results on the validation set seem promising. I wasn’t expecting perfection — my handwriting is too messy for that — but it’s already performing much better than my experience with OneNote, which seems to be one of the best out there for my writing, and the recognition on my reMarkable tablet. I just used the default PyLaia settings in Transkribus so I could probably tune it to be a little better on the current training data, but I’m not sure that would be worth the effort at this point.

Next I have two things I want to do!

Use it! Now I have something working I can switch from transcribing by hand to transcribing with the model and correcting errors. That should allow me to grow my ground truth dataset a bit quicker while also being useful in digitising my notes (and writing blog posts like this one!)
Figure out the tools for myself and reproduce this on my own computer. Except when playing games I rarely use the full power of my desktop so it would be great to be able to put it to some use in the background while I write emails and reports. That would also mean I’d be using a resource I’ve already paid for rather than paying for more Transkribus credits, and give me more flexibility to learn more deeply how this all works by running multiple trainings with different parameters. Finally it would eliminate the need to upload my private notes into the cloud and give mo more peace of mind about their privacy.

Thats all for now, but I look forward to posting more updates!

Vaguely generic update

Thu, 20 Jul 2023 17:17:22 +0100

If it seems like I haven’t posted here much lately then you’re right. Here’s a few bits and pieces I thought I’d share.

Nix & NixOS

I’ve been learning a lot about nix (the package manager) and NixOS (the Linux distribution) lately, and enjoying the consistency and stability it brings. The whole machine’s setup and my user config can all be specified in nixlang and applied reproducibly and consistently on multiple computers. The configuration is declarative: instead of specifying the steps to configure the system (install this package, modify that file) I specify the desired state of the system (these packages are available, these apps are configured like this) and nix updates the actual state accordingly. I can also specify isolated, reproducible environments for individual projects, and deploy services to the cloud, using the same tooling. The language grammar is simple, but the way it’s interpreted takes a bit of getting used to and I’m finding my appreciation of its elegance growing as I understand it more. This website is now built and deployed using nix!

Cool NixOS stuff includes:

Home Manager: apply a declarative configuration to your user as well as your system
nix-doom-emacs: use nix to configure and maintain your Doom emacs setup
Stylix: consistent theming for multiple different programs
deploy-rs: easily configure remote servers using nix

Tabletop role-playing

I’ve been getting into tabletop role-playing games (TTRPGs) at my local games café, initially Dungeons & Dragons 5th edition but increasingly Cypher System games like Numenera. Of course once I get into a thing I get really into the thing, so not only am I playing these games but increasingly acting as GM to run games for others. I attended Collaborations Workshop 2023 earlier in the year, and enjoyed running a game for some friends there one evening.

Twitter archive

Mon, 10 Jul 2023 19:02:14 +0100

Just a quick note to say that since I’ve deleted my Twitter account, I’ve set up a static archive of all my tweets. Yes, even those embarrassing first ones.

This was made possible by Darius Kazemi: https://tinysubversions.com/twitter-archive/make-your-own/

You can still find me on Mastodon as @petrichor:digipres.club or via any of the other links at the bottom of the page.

Request for input: the missing narrative of libraries in research software engineering

Thu, 23 Sep 2021 00:00:00 +0000

Last week I received an email from a friend inviting me to submit a video to a panel 'Missing narratives around diversity and inclusion in Research Software Engineering' hosted as part of SeptembRSE 2021, the goals of which are:

help create inclusive communities in RSE by centring on intersectional voices
increase awareness of power imbalances that negatively impact multiple marginalised groups in research
provide a call-to-action for diversity and inclusion

Obviously I'm always flattered to be asked my opinion, but I wasn't sure what a straight, cisgender, middle-class white man could offer to an event on diversity, so I had a chat with one of the organisers. We agreed that issues of race, gender, sexuality, disability and many more would be well covered by other speakers, and that I would focus on another narrative also frequently missing in discussions of Research Software Engineering (RSE): librarians.

Librarians are a diverse bunch, but we generally have some or more of the following skills, highly valuable to RSE: information management, metadata, digital preservation, teaching/training, communicating across disciplinary boundaries.

Now, I have my own views on this, but this is not my narrative alone, so I would be very grateful for your perspective, dear reader.

Some things that I've thought of so far:

Librarianship is still perceived as a feminine profession, with all the expectations of quiet uncomplaining service that go along with that; this was neatly captured by the wonderful Fobazi Ettarh (@Fobettarh) in the concept of Vocational Awe (see also Vocational Awe and Librarianship: The Lies We Tell Ourselves – In the Library with the Lead Pipe)
Because of that, we're often used only to provide transactional services, but that makes very limited use of our professional knowledge and experience: we're much more effective as collaborators
Librarianship itself is subject to many of the privileges and oppressions you might expect: while more gender-balanced (for example) than many professions, there are way too few librarians of colour, and it gets maler and whiter the closer you get to senior leadership
Some university libraries have gone down the route of employing RSEs themselves. I know that St Andrews did at one point but I'm not sure if that's still the case
The British Library certainly does employ RSEs, as well as collaborating with RSEs in the Alan Turing Institute, for projects like Living With Machines (though we're hardly typical in having active research projects within the Library compared with most academic libraries)
Some (many) dedicated RSE groups are actually really good at collaborating with library colleagues: I was always treated as a peer and collaborator by the RSE team at Sheffield, for example

Please let me know what other perspectives on this I'm missing, I don't want this to just be based on my own biased view. You can chime in via the usual methods: the comments below, via Twitter or Mastodon, via Matrix or by email if you know it. I'll put the video somewhere public when it's done, and try to credit anyone who contributes either in the video or in a followup post.

mxadm: a small CLI Matrix room admin tool

Wed, 04 Aug 2021 17:11:49 +0100

I’ve enjoyed learning Rust (the programming language) recently, but having only really used it for solving programming puzzles I’ve been looking for an excuse to use it for something more practical.

At the same time, I’ve been using and learning about Matrix (the chat/messaging platform), and running some small rooms there I’ve been a bit frustrated that some pretty common admin things don’t have a good user interface in any of the available clients.

So…

I decided to write a little command-line tool to do a few simple tasks, and it’s now released as mxadm! It’s on crates.io, so if you have Rust and cargo available, installing it is as simple as running cargo install mxadm.

I’ve only taught it to do a few things so far:

List your joined rooms
Add/delete a room alias
Tombstone a room (i.e. redirect it to a new room)

I’ll add more as I need them, and I’m open to suggestions too.

It uses matrix-rust-sdk, the Matrix Client-Server SDK for Rust, which is built on the lower-level Ruma library, along with anyhow for error handling. The kind folks in the #matrix-rust-sdk:matrix.org have been particularly kind in helping me get started using it.

More details from:

Suggestions, code reviews, pull requests all welcome, though it will probably take me a while to act on them. Enjoy!

Comments are back

Tue, 29 Jun 2021 20:58:40 +0100

I forgot to mention it at the time, but I’ve added “normal” comments back to the site, as you’ll see below and on most other pages. In place of the Disqus comments I had before I’m now using Cactus Comments, which is open source and self-hostable (though I’m currently not doing that). If you’ve read my previous post about Matrix self-hosting, you might be interested to know that Cactus uses Matrix rooms for data storage and synchronisation and I can moderate and reply to comments directly from my Matrix client.

I’ve still left the webmention code in place too so you can still comment that way either on your own site or via social media.

Intro to the fediverse

Sun, 11 Apr 2021 20:25:45 +0100

Wow, it turns out to be 10 years since I wrote this beginners guide to Twitter. Things have moved on a loooooong way since then.

Far from being the interesting, disruptive technology it was back then, Twitter has become part of the mainstream, the establishment. Almost everyone and everything is on Twitter now, which has both pros and cons.

So what’s the problem?

It’s now possible to follow all sorts of useful information feeds, from live updates on transport delays to your favourite sports team’s play-by-play performance to an almost infinite number of cat pictures. In my professional life it’s almost guaranteed that anyone I meet will be on Twitter, meaning that I can contact them to follow up at a later date without having to exchange contact details (and they have options to block me if they don’t like that).

On the other hand, a medium where everyone’s opinion is equally valid regardless of knowledge or life experience has turned some parts of the internet into a toxic swamp of hatred and vitriol. It’s easier than ever to forget that we have more common ground with any random stranger than we have similarities, and that’s led to some truly awful acts and a poisonous political arena.

Part of the problem here is that each of the social media platforms is controlled by a single entity with almost no accountability to anyone other than shareholders. Technological change has been so rapid that the regulatory regime has no idea how to handle them, leaving them largely free to operate how they want. This has led to a whole heap of nasty consequences that many other people have done a much better job of documenting than I could (Shoshana Zuboff’s book The Age of Surveillance Capitalism is a good example). What I’m going to focus on instead are some possible alternatives.

If you accept the above argument, one obvious solution is to break up the effective monopoly enjoyed by Facebook, Twitter et al. We need to be able to retain the wonderful affordances of social media but democratise control of it, so that it can never be dominated by a small number of overly powerful players.

What’s the solution?

There’s actually a thing that already exists, that almost everyone is familiar with and that already works like this.

It’s email.

There are a hundred thousand email servers, but my email can always find your inbox if I know your address because that address identifies both you and the email service you use, and they communicate using the same protocol, Simple Mail Transfer Protocol (SMTP)¹. I can’t send a message to your Twitter from my Facebook though, because they’re completely incompatible, like oil and water. Facebook has no idea how to talk to Twitter and vice versa (and the companies that control them have zero interest in such interoperability anyway).

Just like email, a federated social media service like Mastodon allows you to use any compatible server, or even run your own, and follow accounts on your home server or anywhere else, even servers running different software as long as they use the same ActivityPub protocol.

There’s no lock-in because you can move to another server any time you like, and interact with all the same people from your new home, just like changing your email address. Smaller servers mean that no one server ends up with enough power to take over and control everything, as the social media giants do with their own platforms. But at the same time, a small server with a small moderator team can enforce local policy much more easily and block accounts or whole servers that host trolls, nazis or other poisonous people.

How do I try it?

I have no problem with anyone for choosing to continue to use what we’re already calling “traditional” social media; frankly, Facebook and Twitter are still useful for me to keep in touch with a lot of my friends. However, I do think it’s useful to know some of the alternatives if only to make a more informed decision to stick with your current choices. Most of these services only ask for an email address when you sign up and use of your real name vs a pseudonym is entirely optional so there’s not really any risk in signing up and giving one a try. That said, make sure you take sensible precautions like not reusing a password from another account.

Instead of…	Try…
Twitter, Facebook	Mastodon, Pleroma, Misskey
Slack, Discord, IRC	Matrix
WhatsApp, FB Messenger, Telegram	Also Matrix
Instagram, Flickr	PixelFed
YouTube	PeerTube
The web	Interplanetary File System (IPFS)

Which, if you can believe it, was formalised nearly 40 years ago in 1982 and has only had fairly minor changes since then! ↩︎

Collaborations Workshop 2021: collaborative ideas & hackday

Wed, 07 Apr 2021 16:24:10 +0100

My last post covered the more “traditional” lectures-and-panel-sessions approach of the first half of the SSI Collaborations Workshop. The rest of the workshop was much more interactive, consisting of a discussion session, a Collaborative Ideas session, and a whole-day hackathon!

The discussion session on day one had us choose a topic (from a list of topics proposed leading up to the workshop) and join a breakout room for that topic with the aim of producing a “speed blog” by then end of 90 minutes. Those speed blogs will be published on the SSI blog over the coming weeks, so I won’t go into that in more detail.

The Collaborative Ideas session is a way of generating hackday ideas, by putting people together at random into small groups to each raise a topic of interest to them before discussing and coming up with a combined idea for a hackday project. Because of the serendipitous nature of the groupings, it’s a really good way of generating new ideas from unexpected combinations of individual interests.

After that, all the ideas from the session, along with a few others proposed by various participants, were pitched as ideas for the hackday and people started to form teams. Not every idea pitched gets worked on during the hackday, but in the end 9 teams of roughly equal size formed to spend the third day working together.

My team’s project: “AHA! An Arts & Humanities Adventure”

There’s a lot of FOMO around choosing which team to join for an event like this: there were so many good ideas and I wanted to work on several of them! In the end I settled on a team developing an escape room concept to help Arts & Humanities scholars understand the benefits of working with research software engineers for their research.

Five of us rapidly mapped out an example storyline for an escape room, got a website set up with GitHub and populated it with the first few stages of the game. We decided to focus on a story that would help the reader get to grips with what an API is and I’m amazed how much we managed to get done in less than a day’s work!

You can try playing through the escape room (so far) yourself on the web, or take a look at the GitHub repository, which contains the source of the website along with a list of outstanding tasks to work on if you’re interested in contributing.

I’m not sure yet whether this project has enough momentum to keep going, but it was a really valuable way both of getting to know and building trust with some new people and demonstrating the concept is worth more work.

Other projects

Here’s a brief rundown of the other projects worked on by teams on the day.

Coding Confessions: Everyone starts somewhere and everyone cuts corners from time to time. Real developers copy and paste! Fight imposter syndrome by looking through some of these confessions or contributing your own. https://coding-confessions.github.io/
CarpenPI: A template to set up a Raspberry Pi with everything you need to run a Carpentries (https://carpentries.org/) data science/software engineering workshop in a remote location without internet access. https://github.com/CarpenPi/docs/wiki
Research Dugnads: A guide to running an event that is a coming together of a research group or team to share knowledge, pass on skills, tidy and review code, among other software and working best practices (based on the Norwegian concept of a dugnad, a form of “voluntary work done together with other people”) https://research-dugnads.github.io/dugnads-hq/
Collaborations Workshop ideas: A meta-project to collect together pitches and ideas from previous Collaborations Workshop conferences and hackdays, to analyse patterns and revisit ideas whose time might now have come. https://github.com/robintw/CW-ideas
howDescribedIs: Integrate existing tools to improve the machine-readable metadata attached to open research projects by integrating projects like SOMEF, codemeta.json and HowFAIRIs (https://howfairis.readthedocs.io/en/latest/index.html). Complete with CI and badges! https://github.com/KnowledgeCaptureAndDiscovery/somef-github-action
Software end-of-project plans: Develop a template to plan and communicate what will happen when the fixed-term project funding for your research software ends. Will maintenance continue? When will the project sunset? Who owns the IP? https://github.com/elichad/software-twilight
Habeas Corpus: A corpus of machine readable data about software used in COVID-19 related research, based on the CORD19 dataset. https://github.com/softwaresaved/habeas-corpus
Credit-all: Extend the all-contributors GitHub bot (https://allcontributors.org/) to include rich information about research project contributions such as the CASRAI Contributor Roles Taxonomy (https://casrai.org/credit/) https://github.com/dokempf/credit-all

I’m excited to see so many metadata-related projects! I plan to take a closer look at what the Habeas Corpus, Credit-all and howDescribedIs teams did when I get time. I also really want to try running a dugnad with my team or for the GLAM Data Science network.

Collaborations Workshop 2021: talks & panel session

Mon, 05 Apr 2021 21:56:09 +0100

I’ve just finished attending (online) the three days of this year’s SSI Collaborations Workshop (CW for short), and once again it’s been a brilliant experience, as well as mentally exhausting, so I thought I’d better get a summary down while it’s still fresh it my mind.

Collaborations Workshop is, as the name suggests, much more focused on facilitating collaborations than a typical conference, and has settled into a structure that starts off with with longer keynotes and lectures, and progressively gets more interactive culminating with a hack day on the third day.

That’s a lot to write about, so for this post I’ll focus on the talks and panel session, and follow up with another post about the collaborative bits. I’ll also probably need to come back and add in more links to bits and pieces once slides and the “official” summary of the event become available.

Updates

2021-04-07 Added links to recordings of keynotes and panel sessions

Provocations

The first day began with two keynotes on this year’s main themes: FAIR Research Software and Diversity & Inclusion, and day 2 had a great panel session focused on disability. All three were streamed live and the recordings remain available on Youtube:

FAIR Research Software

Dr Michelle Barker, Director of the Research Software Alliance, spoke on the challenges to recognition of software as part of the scholarly record: software is not often cited. The FAIR4RS working group has been set up to investigate and create guidance on how the FAIR Principles for data can be adapted to research software as well; as they stand, the Principles are not ideally suited to software. This work will only be the beginning though, as we will also need metrics, training, career paths and much more. ReSA itself has 3 focus areas: people, policy and infrastructure. If you’re interested in getting more involved in this, you can join the ReSA email list.

Equality, Diversity & Inclusion: how to go about it

Dr Chonnettia Jones, Vice President of Research, Michael Smith Foundation for Health Research spoke extensively and persuasively on the need for Equality, Diversity & Inclusion (EDI) initiatives within research, as there is abundant robust evidence that all research outcomes are improved.

She highlighted the difficulties current approaches to EDI have effecting structural change, and changing not just individual behaviours but the cultures & practices that perpetuate iniquity. What initiatives are often constructed around making up for individual deficits, a bitter framing is to start from an understanding of individuals having equal stature but having different tired experiences. Commenting on the current focus on “research excellent” she pointed out that the hyper-competition this promotes is deeply unhealthy. suggesting instead that true excellence requires diversity, and we should focus on an inclusive excellence driven by inclusive leadership.

Equality, Diversity & Inclusion: disability issues

Day 2’s EDI panel session brought together five disabled academics to discuss the problems of disability in research.

Dr Becca Wilson, UKRI Innovation Fellow, Institute of Population Health Science, University of Liverpool (Chair)
Phoenix C S Andrews (PhD Student, Information Studies, University of Sheffield and Freelance Writer)
Dr Ella Gale (Research Associate and Machine Learning Subject Specialist, School of Chemistry, University of Bristol)
Prof Robert Stevens (Professor and Head of Department of Computer Science, University of Manchester)
Dr Robin Wilson (Freelance Data Scientist and SSI Fellow)

NB. The discussion flowed quite freely so the following summary, so the following summary mixes up input from all the panel members.

Researchers are often assumed to be single-minded in following their research calling, and aptness for jobs is often partly judged on “time send”, which disadvantages any disabled person who has been forced to take a career break. On top of this disabled people are often time-poor because of the extra time needed to manage their condition, leaving them with less “output” to show for their time served on many common metrics. This can partially affect early-career researchers, since resources for these are often restricted on a “years-since-PhD” criterion. Time poverty also makes funding with short deadlines that much harder to apply for. Employers add more demands right from the start: new starters are typically expected to complete a health and safety form, generally a brief affair that will suddenly become an 80-page bureaucratic nightmare if you tick the box declaring a disability.

Many employers claim to be inclusive yet utterly fail to understand the needs of their disabled staff. Wheelchairs are liberating for those who use them (despite the awful but common phrase “wheelchair-bound”) and yet employers will refuse to insure a wheelchair while travelling for work, classifying it as a “high value personal item” that the owner would take the same responsibility for as an expensive camera. Computers open up the world for blind people in a way that was never possible without them, but it’s not unusual for mandatory training to be inaccessible to screen readers. Some of these barriers can be overcome, but doing so takes yet more time that could and should be spent on more important work.

What can we do about it? Academia works on patronage whether we like it or not, so be the person who supports people who are different to you rather than focusing on the one you “recognise yourself in” to mentor. As a manager, it’s important to ask each individual what they need and believe them: they are the expert in their own condition and their lived experience of it. Don’t assume that because someone else in your organisation with the same disability needs one set of accommodations, it’s invalid for your staff member to require something totally different. And remember: disability is unusual as a protected characteristic in that anyone can acquire it at any time without warning!

Lightning talks

Lightning talk sessions are always tricky to summarise, and while this doesn’t do them justice, here are a few highlights from my notes.

Data & metadata

Malin Sandstrom talked about a much-needed refinement of contributor role taxonomies for scientific computing
Stephan Druskat showcased a project to crowdsource a corpus of research software for further analysis

Learning & teaching/community

Matthew Bluteau introduced the concept of the “coding dojo” as a way to enhance community of practice. A group of coders got together to practice & learn by working together to solve a problem and explaining their work as they go
- He described 2 models: a code jam, where people work in small groups, and the Randori method, where 2 people do pair programming while the rest observe. I’m excited to try this out!
Steve Crouch talked about intermediate skills and helping people take the next step, which I’m also very interested in with the GLAM Data Science network
Esther Plomp recounted experience of running multiple Carpentry workshops online, while Diego Alonso Alvarez discussed planned workshops on making research software more usable with GUIs
Shoaib Sufi showcased the SSI’s new event organising guide
Caroline Jay reported on a diary study into autonomy & agency in RSE during COVID
- Lopez, T., Jay, C., Wermelinger, M., & Sharp, H. (2021). How has the covid-19 pandemic affected working conditions for research software engineers? Unpublished manuscript.

Wrapping up

That’s not everything! But this post is getting pretty long so I’ll wrap up for now. I’ll try to follow up soon with a summary of the “collaborative” part of Collaborations Workshop: the idea-generating sessions and hackday!

Time for a new look...

Sat, 03 Apr 2021 15:24:05 +0100

I’ve decided to try switching this website back to using Hugo to manage the content and generate the static HTML pages. I’ve been on the Python-based Nikola for a few years now, but recently I’ve been finding it quite slow, and very confusing to understand how to do certain things. I used Hugo recently for the GLAM Data Science Network website and found it had come on a lot since the last time I was using it, so I thought I’d give it another go, and redesign this site to be a bit more minimal at the same time.

The theme is still a work in progress so it’ll probably look a bit rough around the edges for a while, but I think I’m happy enough to publish it now. When I get round to it I might publish some more detailed thoughts on the design.

Ideas for Accessible Communications

Sat, 20 Mar 2021 15:43:47 +0000

The Disability Support Network at work recently ran a survey on “accessible communications”, to develop guidance on how to make communications (especially internal staff comms) more accessible to everyone. I grabbed a copy of my submission because I thought it would be useful to share more widely, so here it is. Please note that these are based on my own experiences only. I am in no way suggesting that these are the only things you would need to do to ensure your communications are fully accessible. They’re just some things to keep in mind.

Policies/procedures/guidance can be stressful to use if anything is vague or inconsistent, or if it looks like there might be more information implied than is explicitly given (a common cause of this is use of jargon in e.g. HR policies). Emails relating to these policies have similar problems, made worse because they tend to be very brief.

Online meetings can be very helpful, but can also be exhausting, especially if there are too many people, or not enough structure. Larger meetings & webinars without agendas (or where the agenda is ignored, or timings are allowed to drift without acknowledgement) are very stressful, as are those where there is not enough structure to ensure fair opportunities to contribute.

Written reference documents and communications should:

Be carefully checked for consistency and clarity
Have all all key points explicitly stated
Explicitly acknowledge the need for flexibility where it is necessary, rather than implying or hinting at it
Clearly define jargon & acronyms where they are necessary to the point being made, and avoid them otherwise
Include links to longer, more explicit versions where space is tight
Provide clear bullet-point summaries with links to the details

Online meetings should:

Include sufficient break time (at least 10 minutes out of every hour) and not allow this to be compromised just because a speaker has misjudged the length of their talk
Include initial “settling-in” time in agendas to avoid timing getting messed up from the start
Ensure the agenda is stuck to, or that divergence from the agenda is acknowledged explicitly by the chair and updated timing briefly discussed to ensure everyone is clear
Establish a norm for participation at the start of the meeting and stick to it e.g. ask people to raise hands when they have a point to make, or have specific time for round-robin contributions
Ensure quiet/introverted people have space to contribute, but don’t force them to do so if they have nothing to add at the time
Offer a text-based alternative to contributing verbally
If appropriate, at the start of the meeting assign specific roles of:
- Gatekeeper: ensures everyone has a chance to contribute
- Timekeeper: ensures meeting runs to time
- Scribe: ensures a consistent record of the meeting
Be chaired by someone with the confidence to enforce the above: offer training to all staff on chairing meetings to ensure everyone has the skills to run a meeting effectively

Matrix self-hosting

Fri, 12 Mar 2021 21:36:12 +0000

I started running my own Matrix server a little while ago. Matrix is something rather cool, a chat system similar to IRC or Slack, but open and federated. Open in that the standard is available for anyone to view, but also the reference implementations of server and client are open source, along with many other clients and a couple of nascent alternative servers. Federated in that, like email, it doesn’t matter what server you sign up with, you can talk to users on your own or any other server.

I decided to host my own for three reasons. Firstly, to see if I could and to learn from it. Secondly, to try and rationalise the Cambrian explosion of Slack teams I was being added to in 2019. Thirdly, to take some control of the loss of access to historical messages in some communities that rely on Slack (especially the Carpentries and RSE communities).

Since then, I’ve also added a fourth goal: taking advantage of various bridges to bring other messaging network I use (such as Signal and Telegram) into a consistent UI. I’ve also found that my use of Matrix-only rooms has grown as more individuals & communities have adopted the platform.

So, I really like Matrix and I use it daily. My problem now is whether to keep self-hosting. Synapse, the only full server implementation at the moment, is really heavy on memory, so I’ve ended up running it on a much bigger server than I thought I’d need, which seems overkill for a single-user instance. So now I have to make a decision about whether it’s worth keeping going, or shutting it down and going back to matrix.org, or setting up on one of the other servers that have sprung up in the last couple of years.

There are a couple of other considerations here. Firstly, Synapse resource usage is entirely down to the size of the rooms joined by users of the homeowner, not directly the number of users. So if users have mostly overlapping interests, and thus keep to the same rooms, you can support quite a large community without significant extra resource usage.

Secondly, there are a couple of alternative server implementations in development specifically addressing this issue for small servers. Dendrite and Conduit. Neither are quite ready for what I want yet, but are getting close, and when ready that will allow running small homeservers with much more sensible resource usage.

So I could start opening up for other users, and at least justify the size of the server that way. I wouldn’t ever want to make it a paid-for service but perhaps people might be willing to make occasional donations towards running costs. That still leaves me with the question of whether I’m comfortable running a service that others may come to rely on, or being responsible for the safety of their information. I could also hold out for Dendrite or Conduit to mature enough that I’m ready to try them, which might not be more than a few months off.

Hmm, seems like I’ve convinced myself to stick with it for now, and we’ll see how it goes. In the meantime, if you know me and you want to try it out let me know and I might risk setting you up with an account!

What do you miss least about pre-lockdown life?

Fri, 26 Feb 2021 21:12:26 +0000

@JanetHughes on Twitter: What do you miss the least from pre-lockdown life?

I absolutely do not miss wandering around the office looking for a meeting room for a confidential call or if I hadn’t managed to book a room in advance. Let’s never return to that joyless frustration, hey?

10:27 AM · Feb 3, 2021

After seeing Terence Eden taking Janet Hughes’ tweet from earlier this month as a writing prompt, I thought I might do the same.

The first thing that leaps to my mind is commuting. At various points in my life I’ve spent between one and three hours a day travelling to and from work and I’ve never more than tolerated it at best. It steals time from your day, and societal norms dictate that it’s your leisure & self-care time that must be sacrificed. Longer commutes allow more time to get into a book or podcast, especially if not driving, but I’d rather have that time at home rather than trying to be comfortable in a train seat designed for some mythical average man shaped nothing like me!

The other thing I don’t miss is the colds and flu! Before the pandemic, British culture encouraged working even when ill, which meant constantly coming into contact with people carrying low-grade viruses. I’m not immunocompromised but some allergies and residue of being asthmatic as a child meant that I would get sick 2-3 times a year. A pleasant side-effect of the COVID precautions we’re all taking is that I haven’t been sick for over 12 months now, which is amazing!

Finally, I don’t miss having so little control over my environment. One of the things that working from home has made clear is that there are certain unavoidable aspects of working in my shared office that cause me sensory stress, and that are completely unrelated to my work. Working (or trying to work) next to a noisy automatic scanner; trying to find a light level that works for 6 different people doing different tasks; lacking somewhere quiet and still to eat lunch and recover from a morning of meetings or the constant vaguely-distracting bustle of a large shared office. It all takes energy. Although it’s partly been replaced by the new stress of living through a global pandemic, that old stress was a constant drain on my productivity and mood that had been growing throughout my career as I moved (ironically, given the common assumption that seniority leads to more privacy) into larger and larger open plan offices.

Remarkable blogging

Sat, 06 Feb 2021 21:15:30 +0000

And the handwritten blog saga continues, as I’ve just received my new reMarkable 2 tablet, which is designed for reading, writing and nothing else. It uses a super-responsive e-ink display and writing on it with a stylus is a dream. It has a slightly rough texture with just a bit of friction that makes my writing come out a lot more legibly than on a slippery glass touchscreen.

If that was all there was to it, I might not have wasted my money, but it turns out that it runs on Linux and the makers have wisely decided not to lock it down but to give you full root mess.

Yes, you read that right: root access. It presents as an ethernet device over USB, so you can SSH in with a password found in the settings and have full control over your own devices. What a novel concept. This fact alone has meant it’s built a small yet devoted community of users who have come up with some clever ways of extending its functionality. In fact, many of these are listed on this GitHub repository.

Finally, from what I’ve seen so far, the handwriting recognition is impressive to say the least. This post was written on it and needed only a little editing. I think this is a device that will get a lot of use!

GLAM Data Science Network fellow travellers

Wed, 03 Feb 2021 16:00:04 +0000

Updates

2021-02-04 Thanks to Gene @dzshuniper@ausglam.space for suggesting ADHO and a better attribution for the opening quote (see comments below for details)

See comments & webmentions for details.

“If you want to go fast, go alone. If you want to go far, go together.” — African proverb, probably popularised in English by Kenyan church leader Rev. Samuel Kobia (original)

This quote is a popular one in the Carpentries community, and I interpret it in this context to mean that a group of people working together is more sustainable than individuals pursuing the same goal independently. That’s something that speaks to me, and that I want to make sure is reflected in nurturing this new community for data science in galleries, archives, libraries & museums (GLAM). To succeed, this work needs to be complementary and collaborative, rather than competitive, so I want to acknowledge a range of other networks & organisations whose activities complement this.

The rest of this article is an unavoidably incomplete list of other relevant organisations whose efforts should be acknowledged and potentially built on. And it should go without saying, but just in case: if the work I’m planning fits right into an existing initiative, then I’m happy to direct my resources there rather than duplicate effort.

Inspirations & collaborators

Groups with similar goals or undertaking similar activities, but focused on a different sector, geographic area or topic. I think we should make as much use of and contribution to these existing communities as possible since there will be significant overlap.

code4lib

Probably the closest existing community to what I want to build, but primarily based in the US, so timezones (and physical distance for in-person events) make it difficult to participate fully. This is a well-established community though, with regular events including an annual conference so there’s a lot to learn here.

newCardigan

Similar to code4lib but an Australian focus, so the timezone problem is even bigger!

GLAM Labs

Focused on supporting the people experimenting with and developing the infrastructure to enable scholars to access GLAM materials in new ways. In some ways, a GLAM data science network would be complementary to their work, by providing people not directly involved with building GLAM Labs with the skills to make best use of GLAM Labs infrastructure.

UK Government data science community

Another existing community with very similar intentions, but focused on UK Government sector. Clearly the British Library and a few national & regional museums & archives fall into this, but much of the rest of the GLAM sector does not.

Artifical Intelligence for Libraries, Archives & Museums (AI4LAM)

A multinational collaboration between several large libraries, archives and museums with a specific focus on the Artificial Intelligence (AI) subset of data science

UK Reproducibility Network

A network of researchers, primarily in HEIs, with an interest in improving the transparency and reliability of academic research. Mostly science-focused but with some overlap of goals around ethical and robust use of data.

Museums Computer Group

I’m less familiar with this than the others, but it seems to have a wider focus on technology generally, within the slightly narrower scope of museums specifically. Again, a lot of potential for collaboration.

Training

Several organisations and looser groups exist specifically to develop and deliver training that will be relevant to members of this network. The network also presents an opportunity for those who have done a workshop with one of these and want to know what the “next steps” are to continue their data science journey.

Supporters

These misson-driven organisations have goals that align well with what I imagine for the GLAM DSN, but operate at a more strategic level. They work by providing expert guidance and policy advice, lobbying and supporting specific projects with funding and/or effort. In particular, the SSI runs a fellowship programme which is currently providing a small amount of funding to this project.

Digital Preservation Coalition (DPC)
Software Sustainability Institute (SSI)
Research Data Alliance (RDA)
Alliance of Digital Humanities Organizations (ADHO)
- … and its Libraries and Digital Humanities Special Interest Group (Lib&DH SIG)

Professional bodies

These organisations exist to promote the interests of professionals in particular fields, including supporting professional development. I hope they will provide communication channels to their various members at the least, and may be interested in supporting more directly, depending on their mission and goals.

Conclusion

As I mentioned at the top of the page, this list cannot possibly be complete. This is a growing area and I’m not the only or first person to have this idea. If you can think of anything glaring that I’ve missed and you think should be on this list, leave a comment or tweet/toot at me!

A new font for the blog

Tue, 26 Jan 2021 09:34:00 +0000

I’ve updated my blog theme to use the quasi-proportional fonts Iosevka Aile and Iosevka Etoile. I really like the aesthetic, as they look like fixed-width console fonts (I use the true fixed-width version of Iosevka in my terminal and text editor) but they’re actually proportional which makes them easier to read.
https://typeof.net/Iosevka/

Training a model to recognise my own handwriting

Mon, 25 Jan 2021 21:08:25 +0000

If I’m going to train an algorithm to read my weird & awful writing, I’m going to need a decent-sized training set to work with. And since one of the main things I want to do with it is to blog “by hand” it makes sense to focus on that type of material for training. In other words, I need to write out a bunch of blog posts on paper, scan them and transcribe them as ground truth. The added bonus of this plan is that after transcribing, I also end up with some digital text I can use as an actual post — multitasking!

So, by the time you read this, I will have already run it through a manual transcription process using Transkribus to add it to my training set, and copy-pasted it into emacs for posting. This is a fun little project because it means I can:

Write more by hand with one of my several nice fountain pens, which I enjoy
Learn more about the operational process some of my colleagues go through when digitising manuscripts
Learn more about the underlying technology & maths, and how to tune the process
Produce more lovely content! For you to read! Yay!
Write in a way that forces me to put off editing until after a first draft is done and focus more on getting the whole of what I want to say down.

That’s it for now — I’ll keep you posted as the project unfolds.

Addendum

Tee hee! I’m actually just enjoying the process of writing stuff by hand in long-form prose. It’ll be interesting to see how the accuracy turns out and if I need to be more careful about neatness. Will it be better or worse than the big but generic models used by Samsung Notes or OneNote. Maybe I should include some stylus-written text for comparison.

Blogging by hand

Fri, 22 Jan 2021 16:16:00 +0000

I wrote the following text on my tablet with a stylus, which was an interesting experience:

So, thinking about ways to make writing fun again, what if I were to write some of them by hand? I mean I have a tablet with a pretty nice stylus, so maybe handwriting recognition could work. One major problem, of course, is that my handwriting is AWFUL! I guess I’ll just have to see whether the OCR is good enough to cope…

It’s something I’ve been thinking about recently anyway: I enjoy writing with a proper fountain pen, so is there a way that I can have a smooth workflow to digitise handwritten text without just typing it back in by hand? That would probably be preferable to this, which actually seems to work quite well but does lead to my hand tensing up to properly control the stylus on the almost-frictionless glass screen.

I’m surprised how well it worked! Here’s a sample of the original text:

And here’s the result of converting that to text with the built-in handwriting recognition in Samsung Notes:

Writing blog posts by hand
So, thinking about ways to make writing fun again, what if I were to write some of chum by hand? I mean, I have a toldest winds a pretty nice stylus, so maybe handwriting recognition could work.
One major problems, ofcourse, is that my , is AWFUL! Iguess
I’ll just have to see whattime the Ocu is good enough to cope…
It’s something I’ve hun tthinking about recently anyway: I enjoy wilting with a proper fountain pion, soischeme a way that I can have a smooch workflow to digitise handwritten text without just typing it back in by hand?
That wouldprobally be preferableto this, which actually scams to work quito wall but doers load to my hand tensing up to properly couldthe stylus once almost-frictionlessg lass scream.

It’s pretty good! It did require a fair bit of editing though, and I reckon we can do better with a model that’s properly trained on a large enough sample of my own handwriting.

What I want from a GLAM/Cultural Heritage Data Science Network

Mon, 18 Jan 2021 17:27:00 +0000

Introduction

As I mentioned last year, I was awarded a Software Sustainability Institute Fellowship to pursue the project of setting up a Cultural Heritage/GLAM data science network. Obviously, the global pandemic has forced a re-think of many plans and this is no exception, so I’m coming back to reflect on it and make sure I’m clear about the core goals so that everything else still moves in the right direction.

One of the main reasons I have for setting up a GLAM data science network is because it’s something I want. The advice to “scratch your own itch” is often given to people looking for an open project to start or contribute to, and the lack of a community of people with whom to learn & share ideas and practice is something that itches for me very much.

The “motivation” section in my original draft project brief for this work said:

Cultural heritage work, like all knowledge work, is increasingly data-based, or at least gives opportunities to make use of data day-to-day. The proper skills to use this data enable more effective working. Knowledge and experience thus gained improves understanding of and empathy with users also using such skills.

But of course, I have my own reasons for wanting to do this too. In particular, I want to:

Advocate for the value of ethical, sustainable data science across a wide range of roles within the British Library and the wider sector
Advance the sector to make the best use of data and digital sources in the most ethical and sustainable way possible
Understand how and why people use data from the British Library, and plan/deliver better services to support that
Keep up to date with relevant developments in data science
Learn from others’ skills and experiences, and share my own in turn

Those initial goals imply some further supporting goals:

Build up the confidence of colleagues who might benefit from data science skills but don’t feel they are “technical” or “computer literate” enough
Further to that, build up a base of colleagues with the confidence to share their skills & knowledge with others, whether through teaching, giving talks, writing or other channels
Identify common awareness gaps (skills/knowledge that people don’t know they’re missing) and address them
Develop a communal space (primarily online) in which people feel safe to ask questions
Develop a body of professional practice and help colleagues to learn and contribute to the evolution of this, including practices of data ethics, software engineering, statistics, high performance computing, …
Break down language barriers between data scientists and others

I’ll expand on this separately as my planning develops, but here are a few specific activities that I’d like to be able to do to support this:

Organise less-formal learning and sharing events to complement the more formal training already available within organisations and the wider sector, including “show and tell” sessions, panel discussions, code cafés, masterclasses, guest speakers, reading/study groups, co-working sessions, …
Organise training to cover intermediate skills and knowledge currently missing from the available options, including the awareness gaps and professional practice mentioned above
Collect together links to other relevant resources to support self-led learning

Decisions to be made

There are all sorts of open questions in my head about this right now, but here are some of the key ones.

Is it GLAM or Cultural Heritage?

When I first started planning this whole thing, I went with “Cultural Heritage”, since I was pretty transparently targeting my own organisation. The British Library is fairly unequivocally a CH organisation. But as I’ve gone along I’ve found myself gravitating more towards the term “GLAM” (which stands for Galleries, Libraries, Archives, Museums) as it covers a similar range of work but is clearer (when you spell out the acronym) about what kinds of work are included.

What skills are relevant?

This turns out to be surprisingly important, at least in terms of how the community is described, as they define the boundaries of the community and can be the difference between someone feeling welcome or excluded. For example, I think that some introductory statistics training would be immensely valuable for anyone working with data to understand what options are open to them and what limitations those options have, but is the word “statistics” offputting per se to those who’ve chosen a career in arts & humanities? I don’t know because I don’t have that background and perspective.

Keep it internal to the BL, or open up early on?

I originally planned to focus primarily on my own organisation to start with, feeling that it would be easier to organise events and build a network within a single organisation. However, the pandemic has changed my thinking significantly. Firstly, it’s now impossible to organise in-person events and that will continue for quite some time to come, so there is less need to focus on the logistics of getting people into the same room. Secondly, people within the sector are much more used to attending remote events, which can easily be opened up to multiple organisations in many countries, timezones allowing. It now makes more sense to focus primarily on online activities, which opens up the possibility of building a critical mass of active participants much more quickly by opening up to the wider sector.

Conclusion

This is the type of post that I could let run and run without ever actually publishing, but since it’s something I need feedback and opinions on from other people, I’d better ship it! I really want to know what you think about this, whether you feel it’s relevant to you and what would make it useful. Comments are open below, or you can contact me via Mastodon or Twitter.

Writing About Not Writing

Fri, 08 Jan 2021 16:11:36 +0000

Under Construction Grunge Sign by Nicolas Raymond — CC BY 2.0

Every year, around this time of year, I start doing two things. First, I start thinking I could really start to understand monads and write more than toy programs in Haskell. This is unlikely to ever actually happen unless and until I get a day job where I can justify writing useful programs in Haskell, but Advent of Code always gets me thinking otherwise.

Second, I start mentally writing this same post. You know, the one about how the blogger in question hasn’t had much time to write but will be back soon?

“Sorry I haven’t written much lately…”

It’s about as cliché as a Geocities site with a permanent “Under construction” GIF. At some point, not long after the dawn of ~~time~~ the internet, most people realised that every website was permanently under construction and publishing something not ready to be published was just pointless.

So I figured this year I’d actually finish writing it and publish it. After all, what’s the worst that could happen?

If we’re getting all reflective about this, I could probably suggest some reasons why I’m not writing much:

For a start, there’s a lot going on in both my world and The World right now, which doesn’t leave a lot of spare energy after getting up, eating, housework, working and a few other necessary activities. As a result, I’m easily distracted and I tend to let myself get dragged off in other directions before I even get to writing much of anything. If I do manage to focus on this blog in general, I’ll often end up working on some minor tweak to the theme or functionality.

I mean, right now I’m wondering if I can do something clever in my text-editor (Emacs, since you’re asking) to streamline my writing & editing process so it’s more elegant, efficient, ergonomic and slightly closer to perfect in every way.

It also makes me much more likely to self-censor, and to indulge my perfectionist tendencies to try and tweak the writing until it’s absolutely perfect, which of course never happens. I’ve got a whole heap of partly-written posts that are juuuust waiting for the right motivation for me to just finish them off.

The only real solution is to accept that:

I’m not going to write much and that’s probably OK
What I do write won’t always be the work of carefully-researched, finely crafted genius that I want it to be, and that’s probably OK too

Also to remember why I started writing and publishing stuff in the first place: to reflect and get my thoughts out onto a (virtual) page so that I can see them, figure out whether I agree with myself and learn; and to stimulate discussion and get other views on my (possibly uninformed, incorrect or half-formed) thoughts, also to learn. In other words, a thing I do for me. It’s easy to forget that and worry too much about whether anyone else wants to read my s—t.

Will you notice any changes? Maybe? Maybe not? Who knows. But it’s a new year and that’s as good a time for a change as any.

Talks

Wed, 21 Oct 2020 14:00:04 +0000

Here is a selection of talks that I’ve given.

{{% template %}} <%! import arrow %>

% for talk in post.data("talks"): % endfor

Date	Title	Location
% if 'date' in talk: <% date = arrow.get(talk['date']) %> ${date.format('ddd d MMM YYYY')} % endif	% if 'url' in talk: % endif ${talk['title']} % if 'url' in talk: % endif	${talk.get('location', '')}

When is a persistent identifier not persistent? Or an identifier?

Tue, 08 Sep 2020 09:19:55 +0100

I wrote a post on the problems with ISBNs as persistent identifiers (PIDS) for work, so check it out if that sounds interesting.

IDCC20 reflections

Tue, 03 Mar 2020 12:21:51 +0000

I’m just back from IDCC20, so here are a few reflections on this year’s conference. You can find all the available slides and links to shared notes on the conference programme. There’s also a list of all the posters and an overview of the Unconference

Skills for curation of diverse datasets

Here in the UK and elsewhere, you’re unlikely to find many institutions claiming to apply a deep level of curation to every dataset/software package/etc deposited with them. There are so many different kinds of data and so few people in any one institution doing “curation” that it’s impossible to do this for everything. Absent the knowledge and skills required to fully evaluate an object the best that can be done is usually to make a sense check on the metadata and flag up with the depositor potential for high-level issues such as accidental disclosure of sensitive personal information.

The Data Curation Network in the United States is aiming to address this issue by pooling expertise across multiple organisations. The pilot has been highly successful and they’re now looking to obtain funding to continue this work. The Swedish National Data Service is experimenting with a similar model, also with a lot of success.

As well as sharing individual expertise, the DCN collaboration has also produced some excellent online quick-reference guides for curating common types of data.

We had some further discussion as part of the Unconference on the final day about what it would look like to introduce this model in the UK. There was general agreement that this was a good idea and a way to make optimal use of sparse resources. There were also very valid concerns that it would be difficult in the current financial climate for anyone to justify doing work for another organisation, apparently for free.

In my mind there are two ways around this, which are not mutually exclusive by any stretch of the imagination. First is to Just Do It: form an informal network of curators around something simple like a mailing list, and give it a try. Second is for one or more trusted organisations to provide some coordination and structure. There are several candidates for this including DCC, Jisc, DPC and the British Library; we all have complementary strengths in this area so it’s my hope that we’ll be able to collaborate around it. In the meantime, I hope the discussion continues.

Artificial intelligence, machine learning et al

As you might expect at any tech-oriented conference there was a strong theme of AI running through many presentations, starting from the very first keynote from Francine Berman. Her talk, The Internet of Things: Utopia or Dystopia? used self-driving cars as a case study to unpack some of the ethical and privacy implications of AI. For example, driverless cars can potentially increase efficiency, both through route-planning and driving technique, but also by allowing fewer vehicles to be shared by more people. However, a shared vehicle is not a private space in the way your own car is: anything you say or do while in that space is potentially open to surveillance.

Aside from this, there are some interesting ideas being discussed, particularly around the possibility of using machine learning to automate increasingly complex actions and workflows such as data curation and metadata enhancement. I didn’t get the impression anyone is doing this in the real world yet, but I’ve previously seen theoretical concepts discussed at IDCC make it into practice so watch this space!

Playing games!

Training is always a major IDCC theme, and this year two of the most popular conference submissions described games used to help teach digital curation concepts and skills.

Mary Donaldson and Matt Mahon of the University of Glasgow presented their use of Lego to teach the concept of sufficient metadata. Participants build simple models before documenting the process and breaking them down again. Then everyone had to use someone else’s documentation to try and recreate the models, learning important lessons about assumptions and including sufficient detail. Kirsty Merrett and Zosia Beckles from the University of Bristol brought along their card game “Researchers, Impact and Publications (RIP)”, based on the popular “Cards Against Humanity”. RIP encourages players to examine some of the reasons for and against data sharing with plenty of humour thrown in. Both games were trialled by many of the attendees during Thursday’s Unconference.

Summary

I realised in Dublin that it’s 8 years since I attended my first IDCC, held at the University of Bristol in December 2011 while I was still working at the nearby University of Bath. While I haven’t been every year, I’ve been to every one held in Europe since then and it’s interesting to see what has and hasn’t changed. We’re no longer discussing data management plans, data scientists or various other things as abstract concepts that we’d like to encourage, but dealing with the real-world consequences of them.

The conference has also grown over the years: this year was the biggest yet, boasting over 300 attendees. There has been especially big growth in attendees from North America, Australasia, Africa and the Middle East. That’s great for the diversity of the conference as it brings in more voices and viewpoints than ever. With more people around to interact with I have to work harder to manage my energy levels but I think that’s a small price to pay.

Iosevka: a nice fixed-width-font

Tue, 28 Jan 2020 11:26:49 +0000

Iosevka is a nice, slender monospace font with a lot of configurable variations. Check it out: https://typeof.net/Iosevka/

SSI Fellowship 2020

Wed, 22 Jan 2020 19:30:00 +0000

I’m honoured and excited to be named one of this year’s Software Sustainability Institute Fellows. There’s not much to write about yet because it’s only just started, but I’m looking forward to sharing more with you. In the meantime, you can take a look at the 2020 fellowship announcement and get an idea of my plans from my application video:

Replacing comments with webmentions

Thu, 16 Jan 2020 20:42:50 +0000

Just a quickie to say that I’ve replaced the comment section at the bottom of each post with webmentions, which allows you to comment by posting on your own site and linking here. It’s a fundamental part of the IndieWeb, which I’m slowly getting to grips with having been a halfway member of it for years by virtue of having my own site on my own domain.

I’d already got rid of Google Analytics to stop forcing that tracking on my visitors, I wanted to get rid of Disqus too because I’m pretty sure the only way that is free for me is if they’re selling my data and yours to third parties. Webmention is a nice alternative because it relies only on open standards, has no tracking and allows people to control their own comments. While I’m currently using a third-party service to help, I can switch to self-hosted at any point in the future, completely transparently.

Thanks to webmention.io, which handles incoming webmentions for me, and webmention.js, which displays them on the site, I can keep it all static and not have to implement any of this myself, which is nice. It’s a bit harder to comment because you have to be able to host your own content somewhere, but then almost no-one ever commented anyway, so it’s not like I’ll lose anything! Plus, if I get Bridgy set up right, you should be able to comment just by replying on Mastodon, Twitter or a few other places.

A spot of web searching shows that I’m not the first to make the Disqus -> webmentions switch (yes, I’m putting these links in blatantly to test outgoing webmentions with Telegraph…):

Let’s see how this goes!

Bridging Carpentries Slack channels to Matrix

Mon, 25 Nov 2019 23:03:50 +0000

It looks like I’ve accidentally taken charge of bridging a bunch of The Carpentries Slack channels over to Matrix. Given this, it seems like a good idea to explain what that sentence means and reflect a little on my reasoning. I’m more than happy to discuss the pros and cons of this approach

If you just want to try chatting in Matrix, jump to the getting started section

What are Slack and Matrix?

Slack (see also on Wikipedia), for those not familiar with it, is an online text chat platform with the feel of IRC (Internet Relay Chat), a modern look and feel and both web and smartphone interfaces. By providing a free tier that meets many peoples’ needs on its own Slack has become the communication platform of choice for thousands of online communities, private projects and more.

One of the major disadvantages of using Slack’s free tier, as many community organisations do, is that as an incentive to upgrade to a paid service your chat history is limited to the most recent 10,000 messages across all channels. For a busy community like The Carpentries, this means that messages older than about 6-7 weeks are already inaccessible, rendering some of the quieter channels apparently empty.

As Slack is at pains to point out, that history isn’t gone, just archived and hidden from view unless you pay the low, low price of $1/user/month. That doesn’t seem too pricy, unless you’re a non-profit organisation with a lot of projects you want to fund and an active membership of several hundred worldwide, at which point it soon adds up. Slack does offer to waive the cost for registered non-profit organisations, but only for one community. The Carpentries is not an independent organisation, but one fiscally sponsored by Community Initiatives, which has already used its free quota of one elsewhere rendering the Carpentries ineligible. Other umbrella organisations such as NumFocus (and, I expect, Mozilla) also run into this problem with Slack.

So, we have a community which is slowly and inexorably losing its own history behind a paywall. For some people this is simply annoying, but from my perspective as a facilitator of the preservation of digital things the community is haemhorraging an important record of its early history.

Enter Matrix.

Matrix is a chat platform similar to IRC, Slack or Discord. It’s divided into separate channels, and users can join one or more of these to take part in the conversation happening in those channels. What sets it apart from older technology like IRC and walled gardens like Slack & Discord is that it’s federated.

Federation means simply that users on any server can communicate with users and channels on any other server. Usernames and channel addresses specify both the individual identifier and the server it calls home, just as your email address contains all the information needed for my email server to route messages to it. While users are currently tied to their home server, channels can be mirrored and synchronised across multiple servers making the overall system much more resilient. Can’t connect to your favourite channel on server X? No problem: just connect via its alias on server Y and when X comes back online it will be resynchronised.

The technology used is much more modern and secure than the aging IRC protocol, and there’s no vender lock-in like there is with closed platforms like Slack and Discord. On top of that, Matrix channels can easily be “bridged” to channels/rooms on other platforms, including, yes, Slack, so that you can join on Matrix and transparently talk to people connected to the bridged room, or vice versa.

So, to summarise:

The current Carpentries Slack channels could be bridged to Matrix at no cost and with no disruption to existing users
The history of those channels from that point on would be retained on matrix.org and accessible even when it’s no longer available on Slack
If at some point in the future The Carpentries chose to invest in its own Matrix server, it could adopt and become the main Matrix home of these channels without disruption to users of either Matrix or (if it’s still in use at that point) Slack
Matrix is an open protocol, with a reference server implementation and wide range of clients all available as free software, which aligns with the values of the Carpentries community

On top of this:

I’m fed up of having so many different Slack teams to switch between to see the channels in all of them, and prefer having all the channels I regularly visit in a single unified interface;
I wanted to see how easy this would be and whether others would also be interested.

Given all this, I thought I’d go ahead and give it a try to see if it made things more manageable for me and to see what the reaction would be from the community.

How can I get started?

!!! reminder Please remember that, like any other Carpentries space, the Code of Conduct applies in all of these channels.

First, sign up for a Matrix account. The quickest way to do this is on the Matrix “Try now” page, which will take you to the Riot Web client which for many is synonymous with Matrix. Other clients are also available for the adventurous.

Second, join one of the channels. The links below will take you to a page that will let you connect via your preferred client. You’ll need to log in as they are set not to allow guest access, but, unlike Slack, you won’t need an invitation to be able to join.

#general — the main open channel to discuss all things Carpentries
#random — anything that would be considered offtopic elsewhere
#welcome — join in and introduce yourself!

That’s all there is to getting started with Matrix. To find all the bridged channels there’s a Matrix “community” that I’ve added them all to: Carpentries Matrix community.

There’s a lot more, including how to bridge your favourite channels from Slack to Matrix, but this is all I’ve got time and space for here! If you want to know more, leave a comment below, or send me a message on Slack (jezcope) or maybe Matrix (@petrichor:matrix.org)! I’ve also made a separate channel for Matrix-Slack discussions: #matrix on Slack and Carpentries Matrix Discussion on Matrix

MozFest19 first reflections

Sun, 24 Nov 2019 21:11:09 +0000

Discussions of neurodiversity at #mozfest Photo by Jennifer Riggins

The other weekend I had my first experience of Mozilla Festival, aka #mozfest.

It was pretty awesome.

I met quite a few people in real life that I’ve previously only known (/stalked) on Twitter, and caught up with others that I haven’t seen for a while.

I had the honour of co-facilitating a workshop session on imposter syndrome and how to deal with it with the wonderful Yo Yehudi and Emmy Tsang. We all learned a lot and hope our participants did too; we’ll be putting together a summary blog post as soon as we can get our act together!

I also attended a great session, led by Kiran Oliver (psst, they’re looking for a new challenge), on how to encourage and support a neurodiverse workforce.

I was only there for the one day, and I really wish that I’d taken the plunge and committed to the whole weekend. There’s always next year though! To be honest, I’m just disappointed that I never had the courage to go sooner,

Music for working

Sat, 26 Oct 2019 19:53:12 +0100

Today¹ the office conversation turned to blocking out background noise. (No, the irony is not lost on me.) Like many people I work in a large, open-plan office, and I’m not alone amongst my colleagues in sometimes needing to find a way to boost concentration by blocking out distractions. Not everyone is like this, but I find music does the trick for me. I also find that different types of music are better for different types of work, and I use this to try and manage my energy better.

There are more distractions than auditory noise, and at times I really struggle with visual noise. Rather than have this post turn into a rant about the evils of open-plan offices, I’ll just mention that the scientific evidence doesn’t paint them in a good light², or at least suggests that the benefits are more limited in scope than is commonly thought³, and move on to what I actually wanted to share: good music for working to.

There are a number of genres that I find useful for working. Generally, these have in common a consistent tempo, a lack of lyrics, and enough variation to prevent boredom without distracting. Familiarity helps my concentration too so I’ll often listen to a restricted set of albums for a while, gradually moving on by dropping one out and bringing in another. In my case this includes:

Traditional dance music, generally from northern and western European traditions for me. This music has to be rhythmically consistent to allow social dancing, and while the melodies are typically simple repeated phrases, skilled musicians improvise around that to make something beautiful. I tend to go through phases of listening to particular traditions; I’m currently listening to a lot of French, Belgian and Scandinavian.
Computer game soundtracks, which are specifically designed to enhance gameplay without distracting, making them perfect for other activities requiring a similar level of concentration.
Chiptunes and other music incorporating it; partly overlapping with the previous category, chiptunes is music made by hacking the audio chips from (usually) old computers and games machines to become an instrument for new music. Because of the nature of the instrument, this will have millisecond-perfect rhythm and again makes for undistracting noise blocking with an extra helping of nostalgia! Purists would disagree with me, but I like artists that combine chiptunes with other instruments and effects to make something more complete-sounding.
Retrowave/synthwave/outrun, synth-driven music that’s instantly familiar as the soundtrack to many 90s sci-fi and thriller movies. Atmospheric, almost dreamy, but rhythmic with a driving beat, it’s another genre that fits into the “pleasing but not too surprising” category for me.

So where to find this stuff?

One of the best resources I’ve found is Music for Programming which provides carefully curated playlists of mostly electronic music designed to energise without distracting. They’re so well done that the tracks move seamlessly, one to the next, without ever getting boring.

Spotify is an obvious option, and I do use it quite a lot. However, I’ve started trying to find ways to support artists more directly, and Bandcamp seems to be a good way of doing that. It’s really easy to browse by genre, or discover artists similar to what you’re currently hearing. You can listen for free as long as you don’t mind occasional nags to buy the music you’re hearing, but you can also buy tracks or albums. Music you’ve paid for is downloadable in several open, DRM-free formats for you to keep, and you know that a decent chunk of that cash is going directly to that artist.

I also love noise generators; not exactly music, but a variety of pleasant background noises, some of which nicely obscure typical office noise. I particularly like mynoise.net, which has a cornucopia of different natural and synthetic noises. Each generator comes with a range of sliders allowing you to tweak the composition and frequency range, and will even animate them randomly for you to create a gently shifting soundscape. A much simpler, but still great, option is Noisli with it’s nice clean interface. Both offer apps for iOS and Android.

For bonus points, you can always try combining one or more of the above. Adding in a noise generator allows me to listen to quieter music while still getting good environmental isolation when I need concentration. Another favourite combo is to open both the cafe and rainfall generators from myNoise, made easier by the ability to pop out a mini-player then open up a second generator.

I must be missing stuff though. What other musical genres should I try? What background sounds are nice to work to?

Well, you know. The other day. Whatever. ↩︎
See e.g.: Lee, So Young, and Jay L. Brand. ‘Effects of Control over Office Workspace on Perceptions of the Work Environment and Work Outcomes’. Journal of Environmental Psychology 25, no. 3 (1 September 2005): 323–33. https://doi.org/10.1016/j.jenvp.2005.08.001. ↩︎
Open plan offices can actually work under certain conditions, The Conversation ↩︎

Working at the British Library: 6 months in

Sun, 05 May 2019 14:45:02 +0100

It barely seems like it, but I’ve been at the British Library now for nearly 6 months. It always takes a long time to adjust and from experience I know it’ll be another year before I feel fully settled, but my team, department and other colleagues have really made me feel welcome and like I belong.

One thing that hasn’t got old yet is the occasional thrill of remembering that I work at my national library now. Every now and then I’ll catch a glimpse of the collections at Boston Spa or step into one of the reading rooms and think “wow, I actually work here!”

I also like having a national and international role to play, which means I get to travel a bit more than I used to. Budgets are still tight so there are limits, and I still prefer to be home more often than not, but there is more scope in this job than I’ve had previously for travelling to conferences, giving talks that change the way people think, and learning in different contexts.

I’m learning a lot too, especially how to work with and manage people split across multiple sites, and the care and feeding of budgets.

As well as missing mo old team at Sheffield, I do also miss some of the direct contact I had with researchers in HE. I especially miss the teaching work, but also the higher-level influencing of more senior academics to change practices on a wider scale. Still, I get to use those influencing skills in different ways now, and I’m still involved with the Carpentries which should let me keep my hand in with teaching.

I still deal with my general tendency to try and do All The Things, and as before I’m slowly learning to recognise it, tame it and very occasionally turn it to my advantage. That also leads to feelings of imposterism that are only magnified by the knowledge that I now work at a national institution! It’s a constant struggle some days to believe that I’ve actually earned my place here through hard work, Even if I don’t always feel that I have, my colleagues here certainly have, so I should have more faith in their opinion of me.

Finally, I couldn’t write this type of thing without mentioning the commute. I’ve gone from 90 minutes each way on a good day (up to twice that if the trains were disrupted) to 35 minutes each way along fairly open roads. I have less time to read, but much more time at home. On top of that, the library has implemented flexitime across all pay grades, with even senior managers strongly encouraged to make full use. Not only is this an important enabler of equality across the organisation, it relieves for me personally the pressure to work over my contracted hours and the guilt I’ve always felt at leaving work even 10 minutes early. If I work late, it’s now a choice I’m making based on business needs instead of guilt and in full knowledge that I’ll get that time back later.

So that’s where I am right now. I’m really enjoying the work and the culture, and I look forward to what the next 6 months will bring!

RDA Plenary 13 reflection

Sat, 13 Apr 2019 21:19:32 +0100

Photo by me

I sit here writing this in the departure lounge at Philadelphia International Airport, waiting for my Aer Lingus flight back after a week at the 13th Research Data Alliance (RDA) Plenary (although I’m actually publishing this a week or so later at home). I’m pretty exhausted, partly because of the jet lag, and partly because it’s been a very full week with so much to take in.

It’s my first time at an RDA Plenary, and it was quite a new experience for me! First off, it’s my first time outside Europe, and thus my first time crossing quite so many timezones. I’ve been waking at 5am and ready to drop by 8pm, but I’ve struggled on through!

Secondly, it’s the biggest conference I’ve been to for a long time, both in number of attendees and number of parallel sessions. There’s been a lot of sustained input so I’ve been very glad to have a room in the conference hotel and be able to escape for a few minutes when I needed to recharge.

Thirdly, it’s not really like any other conference I’ve been to: rather than having large numbers of presentations submitted by attendees, each session comprises lots of parallel meetings of RDA interest groups and working groups. It’s more community-oriented: an opportunity for groups to get together face to face and make plans or show off results.

I found it pretty intense and struggled to take it all in, but incredibly valuable nonetheless. Lots of information to process (I took a lot of notes) and a few contacts to follow up on too, so overall I loved it!

Using Pipfile in Binder

Thu, 14 Mar 2019 20:08:17 +0000

Photo by Sear Greyson on Unsplash

I recently attended a workshop, organised by the excellent team of the Turing Way project, on a tool called BinderHub. BinderHub, along with public hosting platform MyBinder, allows you to publish computational notebooks online as “binders” such that they’re not static but fully interactive. It’s able to do this by using a tool called repo2docker to capture the full computational environment and dependencies required to run the notebook.

!!! aside “What is the Turing Way?” The Turing Way is, in its own words, “a lightly opinionated guide to reproducible data science.” The team is building an open textbook and running a number of workshops for scientists and research software engineers, and you should check out the project on Github. You could even contribute!

The Binder process goes roughly like this:

Do some work in a Jupyter Notebook or similar
Put it into a public git repository
Add some extra metadata describing the packages and versions your code relies on
Go to mybinder.org and tell it where to find your repository
Open the URL it generates for you
Profit

Other than step 5, which can take some time to build the binder, this is a remarkably quick process. It supports a number of different languages too, including built-in support for R, Python and Julia and the ability to configure pretty much any other language that will run on Linux.

However, the Python support currently requires you to have either a requirements.txt or Conda-style environment.yml file to specify dependencies, and I commonly use a Pipfile for this instead. Pipfile allows you to specify a loose range of compatible versions for maximal convenience, but then locks in specific versions for maximal reproducibility. You can upgrade packages any time you want, but you’re fully in control of when that happens, and the locked versions are checked into version control so that everyone working on a project gets consistency.

Since Pipfile is emerging as something of a standard thought I’d see if I could use that in a binder, and it turns out to be remarkably simple. The reference implementation of Pipfile is a tool called pipenv by the prolific Kenneth Reitz. All you need to use this in your binder is two files of one line each.

requirements.txt tells repo2binder to build a Python-based binder, and contains a single line to install the pipenv package:

pipenv

Then postBuild is used by repo2binder to install all other dependencies using pipenv:

pipenv install --system

The --system flag tells pipenv to install packages globally (its default behaviour is to create a Python virtualenv).

With these two files, the binder builds and runs as expected. You can see a complete example that I put together during the workshop here on Gitlab.

What do you think I should write about?

Sat, 09 Mar 2019 09:50:25 +0000

I’ve found it increasingly difficult to make time to blog, and it’s not so much not having the time — I’m pretty privileged in that regard — but finding the motivation. Thinking about what used to motivate me, one of the big things was writing things that other people wanted to read.

Rather than try to guess, I thought I’d ask!

Those who know what I'm about, what would you read about, if it was written by me?

I'm trying to break through the blog-writers block and would love to know what other people would like to see my ill-considered opinions on.
— Jez Cope (@jezcope) March 7, 2019

I’m still looking for ideas, so please tweet me or leave me a comment below. Below are a few thoughts that I’m planning to do something with.

Something taking one of the more techy aspects of Open Research, breaking it down and explaining the benefits for non-techy folks?
— Dr Beth 🏳️‍🌈 🐺 (@PhdGeek) March 7, 2019

Skills (both techy and non techy) that people need to most effectively support RDM
— Kate O'Neill (@KateFONeill) March 7, 2019

Sometimes I forget that my background makes me well-qualified to take some of these technical aspects of the job and break them down for different audiences. There might be a whole series in this…

Carrying on our conversation last week I'd love to hear more about how you've found moving from an HE lib to a national library and how you see the BL's role in RDM. Appreciate this might be a bit niche/me looking for more interesting things to cite :)
— Rosie Higman (@RosieHLib) March 7, 2019

This is interesting, and something I’d like to reflect on; moving from one job to another always has lessons and it’s easy to miss them if you’re not paying attention. Another one for the pile.

Life without admin rights to your computer
— Mike Croucher (@walkingrandomly) March 7, 2019

This is so frustrating as an end user, but at the same time I get that endpoint security is difficult and there are massive risks associated with letting end users have admin rights. This is particularly important at the BL: as custodian’s of a nation’s cultural heritage, the risk for us is bigger than for many and for this reason we are now Cyber Essentials Plus certified. At some point I’d like to do some research and have a conversation with someone who knows a lot more about InfoSec to work out what the proper approach to this, maybe involving VMs and a demilitarized zone on the network.

I’m always looking for more inspiration, so please leave a comment if you’ve got anything you’d like to read my thoughts on. If you’re not familiar with my writing, please take a minute or two to explore the blog; the tags page is probably a good place to get an overview.

Ultimate Hacking Keyboard: first thoughts

Tue, 05 Mar 2019 21:33:48 +0000

Following on from the excitement of having built a functioning keyboard myself, I got a parcel on Monday. Inside was something that I’ve been waiting for since September: an Ultimate Hacking Keyboard! Where the custom-built Laplace is small and quiet for travelling, the UHK is to be my main workhorse in the study at home.

Here are my first impressions:

Key switches: I went with Kailh blue switches from the available options. In stark contrast to the quiet blacks on the Laplace, blues are NOISY! They have an extra piece of plastic inside the switch that causes an audible and tactile click when the switch activates. This makes them very satisfying to type on and should help as I train my fingers not to bottom out while typing, but does make them unsuitable for use in a shared office! Here are some animations showing how the main types of key switch vary.
Layout: This keyboard has what’s known as a 60% layout: no number pad, arrows or function keys. As with the more spartan Laplace, these “missing” keys are made up for with programmable layers. For example, the arrow keys are on the Mod layer on the I/J/K/L keys, so I can access them without moving from the home row. I actually find this preferable to having to move my hand to the right to reach them, and I really never used the number pad in any case.
Split: This is a split keyboard, which means that the left and right halves can be separated to place the hands further apart which eases strain across the shoulders. The UHK has a neat coiled cable joining the two which doesn’t get in the way. A cool design feature is that the two halves can be slotted back together and function perfectly well as a non-split keyboard too, held together by magnets. There are even electrical contacts so that when the two are joined you don’t need the linking cable.
Programming: The board is fully programmable, and this is achieved via a custom (open source) GUI tool which talks to the (open source) firmware on the board. You can have multiple keymaps, each of which has a separate Base, Mod, Fn and Mouse layer, and there’s an LED display that shows a short mnemonic for the currently active map. I already have a customised Dvorak layout for day-to-day use, plus a standard QWERTY for not-me to use and an alternative QWERTY which will be slowly tweaked for games that don’t work well with Dvorak.
Mouse keys: One cool feature that the designers have included in the firmware is the ability to emulate a mouse. There’s a separate layer that allows me to move the cursor, scroll and click without moving my hands from the keyboard.
Palm rests: Not much to say about the palm rests, other than they are solid wood, and chunky, and really add a little something.

I have to say, I really like it so far! Overall it feels really well designed, with every little detail carefully thought out and excellent build quality and a really solid feeling.

Custom-built keyboard

Thu, 21 Feb 2019 16:02:22 +0000

I’m typing this post on a keyboard I made myself, and I’m rather excited about it!

Why make my own keyboard?

I wanted to learn a little bit about practical electronics, and I like to learn by doing
I wanted to have the feeling of making something useful with my own hands
I actually need a small, keyboard with good-quality switches now that I travel a fair bit for work and this lets me completely customise it to my needs
Just because!

While it is possible to make a keyboard completely from scratch, it makes much more sense to put together some premade parts. The parts you need are:

PCB (printed circuit board): the backbone of the keyboard, to which all the other electrical components attach, this defines the possible physical locations for each key
Switches: one for each key to complete a circuit whenever you press it
Keycaps: switches are pretty ugly and pretty uncomfortable to press, so each one gets a cap; these are what you probably think of as the “keys” on your keyboard and come in almost limitless variety of designs (within the obvious size limitation) and are the easiest bit of personalisation
Controller: the clever bit, which detects open and closed switches on the PCB and tells your computer what keys you pressed via a USB cable
Firmware: the program that runs on the controller starts off as source code like any other program, and altering this can make the keyboard behave in loads of different ways, from different layouts to multiple layers accessed by holding a particular key, to macros and even emulating a mouse!

In my case, I’ve gone for the following:

PCB: Laplace from keeb.io, a very compact 47-key (“40%”) board, with no number pad, function keys or number row, but a lot of flexibility for key placement on the bottom row. One of my key design goals was small size so I can just pop it in my bag and have on my lap on the train.
Controller: Elite-C, designed specifically for keyboard builds to be physically compatible with the cheaper Pro Micro, with a more-robust USB port (the Pro Micro’s has a tendency to snap off), and made easier to program with a built-in reset button and better bootloader.
Switches: Gateron Black: Gateron is one of a number of manufacturers of mechanical switches compatible with the popular Cherry range. The black switch is linear (no click or bump at the activation point) and slightly heavier sprung than the more common red. Cherry also make a black switch but the Gateron version is slightly lighter and having tested a few I found them smoother too. My key goal here was to reduce noise, as the stronger spring will help me type accurately without hitting the bottom of the keystroke with an audible sound.
Keycaps: Blank grey PBT in DSA profile: this keyboard layout has a lot of non-standard sized keys, so blank keycaps meant that I wouldn’t be putting lots of keys out of their usual position; they’re also relatively cheap, fairly classy IMHO and a good placeholder until I end up getting some really cool caps on a group buy or something; oh, and it minimises the chance of someone else trying the keyboard and getting freaked out by the layout…
Firmware: QMK (Quantum Mechanical Keyboard), with a work-in-progress layout, based on Dvorak. QMK has a lot of features and allows you to fully program each and every key, with multiple layers accessed through several different routes. Because there are so few keys on this board, I’ll need to make good use of layers to make all the keys on a usual keyboard available. Dvorak Simplified Keyboard

I’m grateful to the folks of the Leeds Hack Space, especially Nav & Mark who patiently coached me in various soldering techniques and good practice, but also everyone else who were so friendly and welcoming and interested in my project.

I’m really pleased with the result, which is small, light and fully customisable. Playing with QMK firmware features will keep me occupied for quite a while! This isn’t the end though, as I’ll need a case to keep the dust out. I’m hoping to be able to 3D print this or mill it from wood with a CNC mill, for which I’ll need to head back to the Hack Space!

Less, but better

Sun, 03 Feb 2019 10:39:31 +0000

“Wenniger aber besser”
— Dieter Rams {:.big-quote}

I can barely believe it’s a full year since I published my intentions for 2018. A lot has happened since then. Principally: in November I started a new job as Data Services Lead at The British Library. One thing that hasn’t changed is my tendency to try to do too much, so this year I’m going to try and focus on a single intention, a translation of designer Dieter Rams’ famous quote above: Less, but better.

This chimes with a couple of other things I was toying with over the Christmas break, as they’re essentially other ways of saying the same thing:

Take it steady
One thing at a time

I’m also going to keep in mind those touchstones from last year:

What difference is this making?
Am I looking after myself?
Do I have evidence for this?

I mainly forget to think about them, so I’ll be sticking up post-its everywhere to help me remember!

How to extend Python with Rust: part 1

Tue, 06 Feb 2018 20:53:52 +0000

Python is great, but I find it useful to have an alternative language under my belt for occasions when no amount of Pythonic cleverness will make some bit of code run fast enough. One of my main reasons for wanting to learn Rust was to have something better than C for that.

Not only does Rust have all sorts of advantages that make it a good choice for code that needs to run fast and correctly, it’s also got a couple of rather nice crates (libraries) that make interfacing with Python a lot nicer.

Here’s a little tutorial to show you how easy it is to call a simple Rust function from Python. If you want to try it yourself, you’ll find the code on GitHub.

!!! prerequisites I’m assuming for this tutorial that you’re already familiar with writing Python scripts and importing & using packages, and that you’re comfortable using the command line. You’ll also need to have installed Rust.

The Rust bit

The quickest way to get compiled code into Python is to use the builtin ctypes package. This is Python’s “Foreign Function Interface” or FFI: a means of calling functions outside the language you’re using to make the call.

ctypes allows us to call arbitrary functions in a shared library¹, as long as those functions conform to certain standard C language calling conventions. Thankfully, Rust tries hard to make it easy for us to build such a shared library.

The first thing to do is to create a new project with cargo, the Rust build tool:

$ cargo new rustfrompy
     Created library `rustfrompy` project

$ tree
.
├── Cargo.toml
└── src
    └── lib.rs

1 directory, 2 files

!!! aside I use the fairly common convention that text set in fixed-width font is either example code or commands to type in. For the latter, a $ precedes the command that you type (omit the $), and lines that don’t start with a $ are output from the previous command. I assume a basic familiarity with Unix-style command line, but I should probably put in some links to resources if you need to learn more!

We need to edit the Cargo.toml file and add a [lib] section:

[package]
name = "rustfrompy"
version = "0.1.0"
authors = ["Jez Cope <j.cope@erambler.co.uk>"]

[dependencies]

[lib]
name = "rustfrompy"
crate-type = ["cdylib"]

This tells cargo that we want to make a C-compatible dynamic library (crate-type = ["cdylib"]) and what to call it, plus some standard metadata. We can then put our code in src/lib.rs.

We’ll just use a simple toy function that adds two numbers together:

#[no_mangle]
pub fn add(a: i64, b: i64) -> i64 {
    a + b
}

Notice the pub keyword, which instructs the compiler to make this function accessible to other modules, and the #[no_mangle] annotation, which tells it to use the standard C naming conventions for functions. If we don’t do this, then Rust will generate a new name for the function for its own nefarious purposes, and as a side effect we won’t know what to call it when we want to use it from Python.

Being good developers, let’s also add a test:

#[cfg(test)]
mod test {
    use ::*;

    #[test]
    fn test_add() {
        assert_eq!(4, add(2, 2));
    }
}

We can now run cargo test which will compile that code and run the test:

$ cargo test
   Compiling rustfrompy v0.1.0 (file:///home/jez/Personal/Projects/rustfrompy)
    Finished dev [unoptimized + debuginfo] target(s) in 1.2 secs
     Running target/debug/deps/rustfrompy-3033caaa9f5f17aa

running 1 test
test test::test_add ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Everything worked! Now just to build that shared library and we can try calling it from Python:

$ cargo build
   Compiling rustfrompy v0.1.0 (file:///home/jez/Personal/Projects/rustfrompy)
    Finished dev [unoptimized + debuginfo] target(s) in 0.30 secs

Notice that the build is unoptimized and includes debugging information: this is useful in development, but once we’re ready to use our code it will run much faster if we compile it with optimisations. Cargo makes this easy:

$ cargo build --release
   Compiling rustfrompy v0.1.0 (file:///home/jez/Personal/Projects/rustfrompy)
    Finished release [optimized] target(s) in 0.30 secs

The Python bit

After all that, the Python bit is pretty short. First we import the ctypes package (which is included in all recent Python versions):

from ctypes import cdll

Cargo has tidied our shared library away into a folder, so we need to tell Python where to load it from. On Linux, it will be called lib<something>.so where the “something” is the crate name from Cargo.toml, “rustfrompy”:

lib = cdll.LoadLibrary('target/release/librustfrompy.so')

Finally we can call the function anywhere we want. Here it is in a pytest-style test:

def test_rust_add():
    assert lib.add(27, 15) == 42

If you have pytest installed (and you should!) you can run the whole test like this:

$ pytest --verbose test.py
====================================== test session starts ======================================
platform linux -- Python 3.6.4, pytest-3.1.1, py-1.4.33, pluggy-0.4.0 -- /home/jez/.virtualenvs/datasci/bin/python
cachedir: .cache
rootdir: /home/jez/Personal/Projects/rustfrompy, inifile:
collected 1 items

test.py::test_rust_add PASSED

It worked! I’ve put both the Rust and Python code on github if you want to try it for yourself.

Shortcomings

Ok, so that was a pretty simple example, and I glossed over a lot of things. For example, what would happen if we did lib.add(2.0, 2)? This causes Python to throw an error because our Rust function only accepts integers (64-bit signed integers, i64, to be precise), and we gave it a floating point number. ctypes can’t guess what type(s) a given function will work with, but it can at least tell us when we get it wrong.

To fix this properly, we need to do some extra work, telling the ctypes library what the argument and return types for each function are. For a more complex library, there will probably be more housekeeping to do, such as translating return codes from functions into more Pythonic-style errors.

For a small example like this there isn’t much of a problem, but the bigger your compiled library the more extra boilerplate is required on the Python side just to use all the functions. When you’re working with an existing library you don’t have much choice about this, but if you’re building it from scratch specifically to interface with Python, there’s a better way using the Python C API. You can call this directly in Rust, but there are a couple of Rust crates that make life much easier, and I’ll be taking a look at those in a future blog post.

.so on Linux, .dylib on Mac and .dll on Windows ↩︎

New Years's irresolution

Sun, 04 Feb 2018 21:26:48 +0000

Photo by Andrew Hughes on Unsplash

I’ve chosen not to make any specific resolutions this year; I’ve found that they just don’t work for me. Like many people, all I get is a sense of guilt when I inevitably fail to live up to the expectations I set myself at the start of the year. However, I have set a couple of what I’m referring to as “themes” for the year: touchstones that I’ll aim to refer to when setting priorities or just feeling a bit overwhelmed or lacking in direction.

They are:

Contribution
Self-care
Measurement

I may do some blog posts expanding on these, but in the meantime, I’ve put together a handful of questions to help me think about priorities and get perspective when I’m doing (or avoiding doing) something.

What difference is this making?
- I feel more motivated when I can figure out how I’m contributing to something bigger than myself.
- In society?
- In my organisation?
- To my friends & family?
Am I looking after myself?
- I focus a lot on the expectations have (or at least that I think others have) of me, but I can’t do anything well unless I’m generally happy and healthy.
- Is this making me happier and healthier?
- Is this building my capacity to to look after myself, my family & friends and do my job?
- Is this worth the amount of time and energy I’m putting in?
Do I have evidence for this?
- I don’t have to base decisions purely on feelings/opinions: I have the skills to obtain, analyse and interpret data.
- Is this fact or opinion? What are the facts?
- Am I overthinking this?
- Can I put a confidence interval for this?

Build documents from code and data with Saga

Fri, 19 Jan 2018 21:37:35 +0000

!!! tldr “TL;DR” I’ve made Saga, a thing for compiling documents by combining code and data with templates.

What is it?

Saga is a very simple command-line tool that reads in one or more data files, runs one or more scripts, then passes the results into a template to produce a final output document. It enables you to maintain a clean separation between data, logic and presentation and produce data-based documents that can easily be updated. That allows the flow of data through the document to be easily understood, a cornerstone of reproducible analysis.

You run it like this:

saga build -d data.yaml -d other_data.yaml \
  -s analysis.py -t report.md.tmpl \
  -O report.md

Any scripts specified with -s will have access to the data in local variables, and any changes to local variables in a script will be retained when everything is passed to the template for rendering.

For debugging, you can also do:

saga dump -d data.yaml -d other_data.yaml -s analysis.py

which will print out the full environment that would be passed to your template with saga build.

Features

Right now this is a really early version. It does the job but I have lots of ideas for features to add if I ever have time. At present it does the following:

Reads data from one or more YAML files
Transforms data with one or more Python scripts
Renders a template in Mako format
Works with any plain-text output format, including Markdown, LaTeX and HTML

Use cases

Write reproducible reports & papers based on machine-readable data
Separate presentation from content in any document, e.g. your CV (example coming soon)
Yours here?

Get it!

I haven’t released this on PyPI yet, but all the code is available on GitHub to try out. If you have pipenv installed (and if you use Python you should!), you can try it out in an isolated virtual environment by doing:

git clone https://github.com/jezcope/sagadoc.git
cd sagadoc
pipenv install
pipenv run saga

or you can set up for development and run some tests:

pipenv install --dev
pipenv run pytest

Why?

Like a lot of people, I have to produce reports for work, often containing statistics computed from data. Although these generally aren’t academic research papers, I see no reason not to aim for a similar level of reproducibility: after all, if I’m telling other people to do it, I’d better take my own advice!

A couple of times now I’ve done this by writing a template that holds the text of the report and placeholders for values, along with a Python script that reads in the data, calculates the statistics I want and completes the template.

This is valuable for two main reasons:

If anyone wants to know how I processed the data and calculated those statistics, it’s all there: no need to try and remember and reproduce a series of button clicks in Excel;
If the data or calculations change, I just need to update the relevant part and run it again, and all the relevant parts of the document will be updated. This is particularly important if changing a single data value requires recalculation of dozens of tables, charts, etc.

It also gives me the potential to factor out and reuse bits of code in the future, add tests and version control everything.

Now that I’ve done this more than once (and it seems likely I’ll do it again) it makes sense to package that script up in a more portable form so I don’t have to write it over and over again (or, shock horror, copy & paste it!). It saves time, and gives others the possibility to make use of it.

Prior art

I’m not the first person to think of this, but I couldn’t find anything that did exactly what I needed.

Several tools will let you interweave code and prose, including the results of evaluating each code snippet in the document: chief among these are Jupyter and Rmarkdown.

There are also tools that let you write code in the order that makes most sense to read and then rearrange it into the right order to execute, so-call literate programming. The original tool for this is the venerable noweb.

Sadly there is very little that combine both of these and allow you to insert the results of various calculations at arbitrary points in a document, independent of the order of either presenting or executing the code. The only two that I’m aware of are: Dexy and org-mode. Unfortunately, Dexy currently only works on Legacy Python (/Python 2) and org-mode requires emacs (which is fine but not exactly portable). Rmarkdown comes close and supports a range of languages but the full feature set is only available with R.

Actually, my ideal solution is org-mode without the emacs dependency, because that’s the most flexible solution; maybe one day I’ll have both the time and skill to implement that. It’s also possible I might be able to figure out Dexy’s internals to add what I want to it, but until then Saga does the job!

Future work

There are lots of features that I’d still like to add when I have time:

Some actual documentation! And examples!
More data formats (e.g. CSV, JSON, TOML)
More languages (e.g. R, Julia)
Fetching remote data over http
Caching of intermediate results to speed up rebuilds

For now, though, I’d love for you to try it out and let me know what you think! As ever, comment here, tweet me or start an issue on GitHub.

Why try Rust for scientific computing?

Thu, 11 Jan 2018 21:31:48 +0000

When you’re writing analysis code, Python (or R, or JavaScript, or …) is usually the right choice. These high-level languages are set up to make you as productive as possible, and common tasks like array manipulation have been well optimised. However, sometimes you just can’t get enough speed and need to turn to a lower-level compiled language. Often that will be C, C++ or Fortran, but I thought I’d do a short post on why I think you should consider Rust.

One of my goals for 2017’s Advent of Code was to learn a modern, memory-safe, statically-typed language. I now know that there are quite a lot of options in this space, but two seem to stand out: Go & Rust. I gave both of them a try, and although I’ll probably go back to give Go a more thorough test at some point I found I got quite hooked on Rust.

Both languages, though young, are definitely production-ready. Servo, the core of the new Firefox browser, is entirely written in Rust. In fact, Mozilla have been trying to rewrite the rendering core in C for nearly a decade, and switching to Rust let them get it done in just a couple of years.

!!! tldr “TL;DR” - It’s fast: competitive with idiomatic C/C++, and no garbage-collection overhead - It’s harder to write buggy code, and compiler errors are actually helpful - It’s C-compatible: you can call into Rust code anywhere you’d call into C, call C/C++ from Rust, and incrementally replace C/C++ code with Rust - It has sensible modern syntax that makes your code clearer and more concise - Support for scientific computing are getting better all the time (matrix algebra libraries, built-in SIMD, safe concurrency) - It has a really friendly and active community - It’s production-ready: Servo, the new rendering core in Firefox, is built entirely in Rust

Performance

To start with, as a compiled language Rust executes much faster than a (pseudo-)interpreted language like Python or R; the price you pay for this is time spent compiling during development. However, having a compile step also allows the language to enforce certain guarantees, such as type-correctness and memory safety, which between them prevent whole classes of bugs from even being possible.

Unlike Go (which, like many higher-level languages, uses a garbage collector), Rust handles memory safety at compile time through the concepts of ownership and borrowing. These can take some getting used to and were a big source of frustration when I was first figuring out the language, but ultimately contribute to Rust’s reliably-fast performance. Performance can be unpredictable in a garbage-collected language because you can’t be sure when the GC is going to run and you need to understand it really well to stand a chance of optimising it if becomes a problem. On the other hand, code that has the potential to be unsafe will result in compilation errors in Rust.

There are a number of benchmarks (example) that show Rust’s performance on a par with idiomatic C & C++ code, something that very few languages can boast.

Helpful error messages

Because beginner Rust programmers often get compile errors, it’s really important that those errors are easy to interpret and fix, and Rust is great at this. Not only does it tell you what went wrong, but wherever possible it prints out your code annotated with arrows to show exactly where the error is, and makes specific suggestions how to fix the error which usually turn out to be correct. It also has a nice suite of warnings (things that don’t cause compilation to fail but may indicate bugs) that are just as informative, and this can be extended even further by using the clippy linting tool to further analyse your code.

warning: unused variable: `y`
 --> hello.rs:3:9
  |
3 |     let y = x;
  |         ^
  |
  = note: #[warn(unused_variables)] on by default
  = note: to avoid this warning, consider using `_y` instead

Easy to integrate with other languages

If you’re like me, you’ll probably only use a low-level language for performance-critical code that you can call from a high-level language, and this is an area where Rust shines. Most programmers will turn to C, C++ or Fortran for this because they have a well established ABI (Application Binary Interface) which can be understood by languages like Python and R¹.

In Rust, it’s trivial to make a C-compatible shared library, and the standard library includes extra features for working with C types. That also means that existing C code can be incrementally ported to Rust: see remacs for an example.

On top of this, there are projects like rust-cpython and PyO3 which provide macros and structures that wrap the Python C API to let you build Python modules in Rust with minimal glue code; rustr does a similar job for R.

Nice language features

Rust has some really nice features, which let you write efficient, concise and correct code. Several feel particularly comfortable as they remind me of similar things available in Haskell, including:

Enums, a super-powered combination of C enums and unions (similar to Haskell’s algebraic data types) that enable some really nice code with no runtime cost
Generics and traits that let you get more done with less code
Pattern matching, a kind of case statement that lets you extract parts of structs, tuples & enums and do all sorts of other clever things
Lazy computation based on an iterator pattern, for efficient processing of lists of things: you can do for item in list { ... } instead of the C-style use of an index², or you can use higher-order functions like map and filter
Functions/closures as first-class citizens

Scientific computing

Although it’s a general-purpose language and not designed specifically for scientific computing, Rust’s support is improving all the time. There are some interesting matrix algebra libraries available, and built-in SIMD is incoming. The memory safety features also work to ensure thread safety, so it’s harder to write concurrency bugs. You should be able to use your favourite MPI implementation too, and there’s at least one attempt to portably wrap MPI in a more Rust-like way.

Active development and friendly community

One of the things you notice straight away is how active and friendly the Rust community is. There are several IRC channels on irc.mozilla.org including #rust-beginners, which is a great place to get help. The compiler is under constant but carefully-managed development, so that new features are landing all the time but without breaking existing code. And the fabulous Cargo build tool and crates.io are enabling the rapid growth of a healthy ecosystem of open source libraries that you can use to write less code yourself.

Summary

So, next time you need a compiled language to speed up hotspots in your code, try Rust. I promise you won’t regret it!

Julia actually allows you to call C and Fortran functions as a first-class language feature ↩︎
Actually, since C++11 there’s for (auto item : list) { ... } but still… ↩︎

Reflections on #aoc2017

Tue, 02 Jan 2018 21:02:37 +0000

Trees reflected in a lake Joshua Reddekopp on Unsplash

It seems like ages ago, but way back in November I committed to completing Advent of Code. I managed it all, and it was fun!

All of my code is available on GitHub if you’re interested in seeing what I did, and I managed to get out a blog post for every one with a bit more commentary, which you can see in the series list above.

How did I approach it?

I’ve not really done any serious programming challenges before. I don’t get to write a lot of code at the moment, so all I wanted from AoC was an excuse to do some proper problem-solving.

I never really intended to take a polyglot approach, though I did think that I might use mainly Python with a bit of Haskell. In the end, though, I used: Python (×12); Haskell (×7); Rust (×4); Go; C++; Ruby; Julia; and Coconut.

For the most part, my priorities were getting the right answer, followed by writing readable code. I didn’t specifically focus on performance but did try to avoid falling into traps that I knew about.

What did I learn?

I found Python the easiest to get on with: it’s the language I know best and although I can’t always remember exact method names and parameters I know what’s available and where to look to remind myself, as well as most of the common idioms and some performance traps to avoid. Python was therefore the language that let me focus most on solving the problem itself. C++ and Ruby were more challenging, and it was harder to write good idiomatic code but I can still remember quite a lot.

Haskell I haven’t used since university, and just like back then I really enjoyed working out how to solve problems in a functional style while still being readable and efficient (not always something I achieved…). I learned a lot about core Haskell concepts like monads & functors, and I’m really amazed by the way the Haskell community and ecosystem has grown up in the last decade.

I also wanted to learn at least one modern, memory-safe compiled language, so I tried both Go and Rust. Both seem like useful languages, but Rust really intrigued me with its conceptual similarities to both Haskell and C++ and its promise of memory safety without a garbage collector. I struggled a lot initially with the “borrow checker” (the component that enforces memory safety at compile time) but eventually started thinking in terms of ownership and lifetimes after which things became easier. The Rust community seems really vibrant and friendly too.

What next?

I really want to keep this up, so I’m going to look out some more programming challenges (Project Euler looks interesting). It turns out there’s a regular Code Dojo meetup in Leeds, so hopefully I’ll try that out too. I’d like to do more realistic data-science stuff, so I’ll be taking a closer look at stuff like Kaggle too, and figuring out how to do a bit more analysis at work.

I’m also feeling motivated to find an open source project to contribute to and/or release a project of my own, so we’ll see if that goes anywhere! I’ve always found the advice to “scratch your own itch” difficult to follow because everything I think of myself has already been done better. Most of the projects I use enough to want to contribute to tend to be pretty well developed with big communities and any bugs that might be accessible to me will be picked off and fixed before I have a chance to get started. Maybe it’s time to get over myself and just reimplement something that already exists, just for the fun of it!

The Halting Problem — Python — #adventofcode Day 25

Tue, 02 Jan 2018 20:47:19 +0000

Today’s challenge, takes us back to a bit of computing history: a good old-fashioned Turing Machine.

→ Full code on GitHub

!!! commentary Today’s challenge was a nice bit of nostalgia, taking me back to my university days learning about the theory of computing. Turing Machines are a classic bit of computing theory, and are provably able to compute any value that is possible to compute: a value is computable if and only if a Turing Machine can be written that computes it (though in practice anything non-trivial is mind-bendingly hard to write as a TM).

A bit of a library-fest today, compared to other days!

from collections import deque, namedtuple
from collections.abc import Iterator
from tqdm import tqdm
import re
import fileinput as fi

These regular expressions are used to parse the input that defines the transition table for the machine.

RE_ISTATE = re.compile(r'Begin in state (?P<state>\w+)\.')
RE_RUNTIME = re.compile(
    r'Perform a diagnostic checksum after (?P<steps>\d+) steps.')
RE_STATETRANS = re.compile(
    r"In state (?P<state>\w+):\n"
    r"  If the current value is (?P<read0>\d+):\n"
    r"    - Write the value (?P<write0>\d+)\.\n"
    r"    - Move one slot to the (?P<move0>left|right).\n"
    r"    - Continue with state (?P<next0>\w+).\n"
    r"  If the current value is (?P<read1>\d+):\n"
    r"    - Write the value (?P<write1>\d+)\.\n"
    r"    - Move one slot to the (?P<move1>left|right).\n"
    r"    - Continue with state (?P<next1>\w+).")
MOVE = {'left': -1, 'right': 1}

A namedtuple to provide some sugar when using a transition rule.

Rule = namedtuple('Rule', 'write move next_state')

The TuringMachine class does all the work.

class TuringMachine:
    def __init__(self, program=None):
        self.tape = deque()
        self.transition_table = {}
        self.state = None
        self.runtime = 0
        self.steps = 0
        self.pos = 0
        self.offset = 0

        if program is not None:
            self.load(program)

    def __str__(self):
        return f"Current: {self.state}; steps: {self.steps} of {self.runtime}"

Some jiggery-pokery to allow us to use self[pos] to reference an infinite tape.

    def __getitem__(self, i):
        i += self.offset
        if i < 0 or i >= len(self.tape):
            return 0
        else:
            return self.tape[i]

    def __setitem__(self, i, x):
        i += self.offset
        if i >= 0 and i < len(self.tape):
            self.tape[i] = x
        elif i == -1:
            self.tape.appendleft(x)
            self.offset += 1
        elif i == len(self.tape):
            self.tape.append(x)
        else:
            raise IndexError('Tried to set position off end of tape')

Parse the program and set up the transtion table.

    def load(self, program):
        if isinstance(program, Iterator):
            program = ''.join(program)

        match = RE_ISTATE.search(program)
        self.state = match['state']

        match = RE_RUNTIME.search(program)
        self.runtime = int(match['steps'])

        for match in RE_STATETRANS.finditer(program):
            self.transition_table[match['state']] = {
                int(match['read0']): Rule(write=int(match['write0']),
                                          move=MOVE[match['move0']],
                                          next_state=match['next0']),
                int(match['read1']): Rule(write=int(match['write1']),
                                          move=MOVE[match['move1']],
                                          next_state=match['next1']),
            }

Run the program for the required number of steps (given by self.runtime). tqdm isn’t in the standard library but it should be: it shows a lovely text-mode progress bar as we go.

    def run(self):
        for _ in tqdm(range(self.runtime),
                      desc="Running", unit="steps", unit_scale=True):
            read = self[self.pos]
            rule = self.transition_table[self.state][read]
            self[self.pos] = rule.write
            self.pos += rule.move
            self.state = rule.next_state

Calculate the “diagnostic checksum” required for the answer.

    @property
    def checksum(self):
        return sum(self.tape)

Aaand GO!

machine = TuringMachine(fi.input())
machine.run()
print("Checksum:", machine.checksum)

Electromagnetic Moat — Rust — #adventofcode Day 24

Sun, 24 Dec 2017 20:11:14 +0000

Today’s challenge, the penultimate, requires us to build a bridge capable of reaching across to the CPU, our final destination.

→ Full code on GitHub

!!! commentary We have a finite number of components that fit together in a restricted way from which to build a bridge, and we have to work out both the strongest and the longest bridge we can build. The most obvious way to do this is to recursively build every possible bridge and select the best, but that’s an O(n!) algorithm that could blow up quickly, so might as well go with a nice fast language! Might have to try this in Haskell too, because it’s the type of algorithm that lends itself naturally to a pure functional approach.

I feel like I've applied some of the things I've learned in previous challenges I used Rust for, and spent less time mucking about with ownership, and made better use of various language features, including structs and iterators. I'm rather pleased with how my learning of this language is progressing. I'm definitely overusing `Option.unwrap` at the moment though: this is a lazy way to deal with `Option` results and will panic if the result is not what's expected. I'm not sure whether I need to be cloning the components `Vector` either, or whether I could just be passing iterators around.

First, we import some bits of standard library and define some data types. The BridgeResult struct lets us use the same algorithm for both parts of the challenge and simply change the value used to calculate the maximum.

use std::io;
use std::fmt;
use std::io::BufRead;

#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
struct Component(u8, u8);

#[derive(Debug, Copy, Clone, Default)]
struct BridgeResult {
    strength: u16,
    length: u16,
}

impl Component {
    fn from_str(s: &str) -> Component {
        let parts: Vec<&str> = s.split('/').collect();
        assert!(parts.len() == 2);
        Component(parts[0].parse().unwrap(), parts[1].parse().unwrap())
    }

    fn fits(self, port: u8) -> bool {
        self.0 == port || self.1 == port
    }

    fn other_end(self, port: u8) -> u8 {
        if self.0 == port {
            return self.1;
        } else if self.1 == port {
            return self.0;
        } else {
            panic!("{} doesn't fit port {}", self, port);
        }
    }

    fn strength(self) -> u16 {
        self.0 as u16 + self.1 as u16
    }
}

impl fmt::Display for BridgeResult {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "(S: {}, L: {})", self.strength, self.length)
    }
}

best_bridge calculates the length and strength of the “best” bridge that can be built from the remaining components and fits the required port. Whether this is based on strength or length is given by the key parameter, which is passed to Iter.max_by_key.

fn best_bridge<F>(port: u8, key: &F, components: &Vec<Component>) -> Option<BridgeResult>
    where F: Fn(&BridgeResult) -> u16
{
    if components.len() == 0 {
        return None;
    }
    
    components.iter()
        .filter(|c| c.fits(port))
        .map(|c| {
            let b = best_bridge(c.other_end(port), key,
                                &components.clone().into_iter()
                                .filter(|x| x != c).collect())
                .unwrap_or_default();
             BridgeResult{strength: c.strength() + b.strength,
                          length: 1 + b.length}
        })
        .max_by_key(key)
}

Now all that remains is to read the input and calculate the result. I was rather pleasantly surprised to find that in spite of my pessimistic predictions about efficiency, when compiled with optimisations turned on this terminates in less than 1s on my laptop.

fn main() {
    let stdin = io::stdin();
    let components: Vec<_> = stdin.lock()
        .lines()
        .map(|l| Component::from_str(&l.unwrap()))
        .collect();

    match best_bridge(0, &|b: &BridgeResult| b.strength, &components) {
        Some(b) => println!("Strongest bridge is {}", b),
        None => println!("No strongest bridge found")
    };
    match best_bridge(0, &|b: &BridgeResult| b.length, &components) {
        Some(b) => println!("Longest bridge is {}", b),
        None => println!("No longest bridge found")
    };
}

Coprocessor Conflagration — Haskell — #adventofcode Day 23

Sun, 24 Dec 2017 19:47:43 +0000

Today’s challenge requires us to understand why a coprocessor is working so hard to perform an apparently simple calculation.

→ Full code on GitHub

!!! commentary Today’s problem is based on an assembly-like language very similar to day 18, so I went back and adapted my code from that, which works well for the first part. I’ve also incorporated some advice from /r/haskell, and cleaned up all warnings shown by the -Wall compiler flag and the hlint tool.

Part 2 requires the algorithm to run with much larger inputs, and since some analysis shows that it's an `O(n^3)` algorithm it gets intractible pretty fast. There are several approaches to this. First up, if you have a fast enough processor and an efficient enough implementation I suspect that the simulation would probably terminate eventually, but that would likely still take hours: not good enough. I also thought about doing some peephole optimisations on the instructions, but the last time I did compiler optimisation was my degree so I wasn't really sure where to start. What I ended up doing was actually analysing the input code by hand to figure out what it was doing, and then just doing that calculation in a sensible way. I'd like to say I managed this on my own (and I ike to think I would have) but I did get some tips on [/r/adventofcode](https://reddit.com/r/adventofcode).

The majority of this code is simply a cleaned-up version of day 18, with some tweaks to accommodate the different instruction set:

module Main where

import qualified Data.Vector as V
import qualified Data.Map.Strict as M
import Control.Monad.State.Strict
import Text.ParserCombinators.Parsec hiding (State)

type Register = Char
type Value = Int
type Argument = Either Value Register
data Instruction = Set Register Argument
                 | Sub Register Argument
                 | Mul Register Argument
                 | Jnz Argument Argument
  deriving Show
type Program = V.Vector Instruction
data Result = Cont | Halt deriving (Eq, Show)

type Registers = M.Map Char Int
data Machine = Machine { dRegisters :: Registers
                       , dPtr :: !Int
                       , dMulCount :: !Int
                       , dProgram :: Program }

instance Show Machine where
  show d = show (dRegisters d) ++ " @" ++ show (dPtr d) ++ " ×" ++ show (dMulCount d)

defaultMachine :: Machine
defaultMachine = Machine M.empty 0 0 V.empty

type MachineState = State Machine

program :: GenParser Char st Program
program = do
  instructions <- endBy instruction eol
  return $ V.fromList instructions
  where
    instruction = try (regOp "set" Set) <|> regOp "sub" Sub
                  <|> regOp "mul" Mul <|> jump "jnz" Jnz
    regOp n c = do
      string n >> spaces
      val1 <- oneOf "abcdefgh"
      secondArg c val1
    jump n c = do
      string n >> spaces
      val1 <- regOrVal
      secondArg c val1
    secondArg c val1 = do
      spaces
      val2 <- regOrVal
      return $ c val1 val2
    regOrVal = register <|> value
    register = do
      name <- lower
      return $ Right name
    value = do
      val <- many $ oneOf "-0123456789"
      return $ Left $ read val
    eol = char '\n'

parseProgram :: String -> Either ParseError Program
parseProgram = parse program ""

getReg :: Char -> MachineState Int
getReg r = do
  st <- get
  return $ M.findWithDefault 0 r (dRegisters st)

putReg :: Char -> Int -> MachineState ()
putReg r v = do
  st <- get
  let current = dRegisters st
      new = M.insert r v current
  put $ st { dRegisters = new }

modReg :: (Int -> Int -> Int) -> Char -> Argument -> MachineState ()
modReg op r v = do
  u <- getReg r
  v' <- getRegOrVal v
  putReg r (u `op` v')
  incPtr

getRegOrVal :: Argument -> MachineState Int
getRegOrVal = either return getReg

addPtr :: Int -> MachineState ()
addPtr n = do
  st <- get
  put $ st { dPtr = n + dPtr st }

incPtr :: MachineState ()
incPtr = addPtr 1

execInst :: Instruction -> MachineState ()
execInst (Set reg val) = do
  newVal <- getRegOrVal val
  putReg reg newVal
  incPtr
execInst (Mul reg val) = do
  result <- modReg (*) reg val
  st <- get
  put $ st { dMulCount = 1 + dMulCount st }
  return result
execInst (Sub reg val) = modReg (-) reg val
execInst (Jnz val1 val2) = do
  test <- getRegOrVal val1
  jump <- if test /= 0 then getRegOrVal val2 else return 1
  addPtr jump

execNext :: MachineState Result
execNext = do
  st <- get
  let prog = dProgram st
      p = dPtr st
  if p >= length prog then return Halt else do
    execInst (prog V.! p)
    return Cont

runUntilTerm :: MachineState ()
runUntilTerm = do
  result <- execNext
  unless (result == Halt) runUntilTerm

This implements the actual calculation: the number of non-primes between (for my input) 107900 and 124900:

optimisedCalc :: Int -> Int -> Int -> Int
optimisedCalc a b k = sum $ map (const 1) $ filter notPrime [a,a+k..b]
  where
    notPrime n = elem 0 $ map (mod n) [2..(floor $ sqrt (fromIntegral n :: Double))]

main :: IO ()
main = do
  input <- getContents  
  case parseProgram input of
    Right prog -> do
      let c = defaultMachine { dProgram = prog }
          (_, c') = runState runUntilTerm c
      putStrLn $ show (dMulCount c') ++ " multiplications made"
      putStrLn $ "Calculation result: " ++ show (optimisedCalc 107900 124900 17)
    Left e -> print e

Sporifica Virus — Rust — #adventofcode Day 22

Sun, 24 Dec 2017 19:32:03 +0000

Today’s challenge has us helping to clean up (or spread, I can’t really tell) an infection of the “sporifica” virus.

→ Full code on GitHub

!!! commentary I thought I’d have another play with Rust, as its Haskell-like features resonate with me at the moment. I struggled quite a lot with the Rust concepts of ownership and borrowing, and this is a cleaned-up version of the code based on some good advice from the folks on /r/rust.

use std::io;
use std::env;
use std::io::BufRead;
use std::collections::HashMap;

#[derive(PartialEq, Clone, Copy, Debug)]
enum Direction {Up, Right, Down, Left}
#[derive(PartialEq, Clone, Copy, Debug)]
enum Infection {Clean, Weakened, Infected, Flagged}

use self::Direction::*;
use self::Infection::*;

type Grid = HashMap<(isize, isize), Infection>;

fn turn_left(d: Direction) -> Direction {
    match d {Up => Left, Right => Up, Down => Right, Left => Down}
}

fn turn_right(d: Direction) -> Direction {
    match d {Up => Right, Right => Down, Down => Left, Left => Up}
}

fn turn_around(d: Direction) -> Direction {
    match d {Up => Down, Right => Left, Down => Up, Left => Right}
}

fn make_move(d: Direction, x: isize, y: isize) -> (isize, isize) {
    match d {
        Up    => (x-1, y),
        Right => (x, y+1),
        Down  => (x+1, y),
        Left  => (x, y-1),
    }
}

fn basic_step(grid: &mut Grid, x: &mut isize, y: &mut isize, d: &mut Direction) -> usize {
    let mut infect = 0;
    let current = match grid.get(&(*x, *y)) {
        Some(v) => *v,
        None => Clean,
    };
    if current == Infected {
        *d = turn_right(*d);
    } else {
        *d = turn_left(*d);
        infect = 1;
    };
    grid.insert((*x, *y), match current {
        Clean => Infected,
        Infected => Clean,
        x => panic!("Unexpected infection state {:?}", x),
    });
    let new_pos = make_move(*d, *x, *y);
    *x = new_pos.0;
    *y = new_pos.1;
    
    infect
}

fn nasty_step(grid: &mut Grid, x: &mut isize, y: &mut isize, d: &mut Direction) -> usize {
    let mut infect = 0;
    let new_state: Infection;
    let current = match grid.get(&(*x, *y)) {
        Some(v) => *v,
        None => Infection::Clean,
    };
    match current {
        Clean => {
            *d = turn_left(*d);
            new_state = Weakened;
        },
        Weakened => {
            new_state = Infected;
            infect = 1;
        },
        Infected => {
            *d = turn_right(*d);
            new_state = Flagged;
        },
        Flagged => {
            *d = turn_around(*d);
            new_state = Clean;
        }
    };
    grid.insert((*x, *y), new_state);
    let new_pos = make_move(*d, *x, *y);
    *x = new_pos.0;
    *y = new_pos.1;
    
    infect
}

fn virus_infect<F>(mut grid: Grid, mut step: F, mut x: isize, mut y: isize, mut d: Direction, n: usize) -> usize
    where F: FnMut(&mut Grid, &mut isize, &mut isize, &mut Direction) -> usize,
{
    (0..n).map(|_| step(&mut grid, &mut x, &mut y, &mut d))
        .sum()
}

fn main() {
    let args: Vec<String> = env::args().collect();
    let n_basic: usize = args[1].parse().unwrap();
    let n_nasty: usize = args[2].parse().unwrap();
    
    let stdin = io::stdin();
    let lines: Vec<String> = stdin.lock()
        .lines()
        .map(|x| x.unwrap())
        .collect();
    let mut grid: Grid = HashMap::new();
    let x0 = (lines.len() / 2) as isize;
    let y0 = (lines[0].len() / 2) as isize;

    for (i, line) in lines.iter().enumerate() {
        for (j, c) in line.chars().enumerate() {
            grid.insert((i as isize, j as isize),
                        match c {'#' => Infected, _ => Clean});
        }
    }

    let basic_steps = virus_infect(grid.clone(), basic_step, x0, y0, Up, n_basic);
    println!("Basic: infected {} times", basic_steps);

    let nasty_steps = virus_infect(grid, nasty_step, x0, y0, Up, n_nasty);
    println!("Nasty: infected {} times", nasty_steps);
}

Fractal Art — Python — #adventofcode Day 21

Sun, 24 Dec 2017 19:21:01 +0000

Today’s challenge asks us to assist an artist building fractal patterns from a rulebook.

→ Full code on GitHub

!!! commentary Another fairly straightforward algorithm: the really tricky part was breaking the pattern up into chunks and rejoining it again. I could probably have done that more efficiently, and would have needed to if I had to go for a few more iterations and the grid grows with every iteration and gets big fast.

Still behind on the blog posts…

import fileinput as fi
from math import sqrt
from functools import reduce, partial
import operator

INITIAL_PATTERN = ((0, 1, 0), (0, 0, 1), (1, 1, 1))
DECODE = ['.', '#']
ENCODE = {'.': 0, '#': 1}

concat = partial(reduce, operator.concat)


def rotate(p):
    size = len(p)
    return tuple(tuple(p[i][j] for i in range(size))
                 for j in range(size - 1, -1, -1))


def flip(p):
    return tuple(p[i] for i in range(len(p) - 1, -1, -1))


def permutations(p):
    yield p
    yield flip(p)
    for _ in range(3):
        p = rotate(p)
        yield p
        yield flip(p)


def print_pattern(p):
    print('-' * len(p))
    for row in p:
        print(' '.join(DECODE[x] for x in row))
    print('-' * len(p))


def build_pattern(s):
    return tuple(tuple(ENCODE[c] for c in row)
                 for row in s.split('/'))


def build_pattern_book(lines):
    book = {}
    for line in lines:
        source, target = line.strip().split(' => ')
        for rotation in permutations(build_pattern(source)):
            book[rotation] = build_pattern(target)

    return book


def subdivide(pattern):
    size = 2 if len(pattern) % 2 == 0 else 3
    n = len(pattern) // size
    return (tuple(tuple(pattern[i][j] for j in range(y * size, (y + 1) * size))
                  for i in range(x * size, (x + 1) * size))
            for x in range(n)
            for y in range(n))


def rejoin(parts):
    n = int(sqrt(len(parts)))
    size = len(parts[0])
    return tuple(concat(parts[i + k][j] for i in range(n))
                 for k in range(0, len(parts), n)
                 for j in range(size))


def enhance_once(p, book):
    return rejoin(tuple(book[part] for part in subdivide(p)))


def enhance(p, book, n, progress=None):
    for _ in range(n):
        p = enhance_once(p, book)
    return p


book = build_pattern_book(fi.input())

intermediate_pattern = enhance(INITIAL_PATTERN, book, 5)
print("After 5 iterations:", sum(sum(row) for row in intermediate_pattern))

final_pattern = enhance(intermediate_pattern, book, 13)
print("After 18 iterations:", sum(sum(row) for row in final_pattern))

Particle Swarm — Python — #adventofcode Day 20

Sun, 24 Dec 2017 19:13:51 +0000

Today’s challenge finds us simulating the movements of particles in space.

→ Full code on GitHub

!!! commentary Back to Python for this one, another relatively straightforward simulation, although it’s easier to calculate the answer to part 1 than to simulate.

import fileinput as fi
import numpy as np
import re

First we parse the input into 3 2D arrays: using numpy enables us to do efficient arithmetic across the whole set of particles in one go.

PARTICLE_RE = re.compile(r'p=<(-?\d+),(-?\d+),(-?\d+)>, '
                         r'v=<(-?\d+),(-?\d+),(-?\d+)>, '
                         r'a=<(-?\d+),(-?\d+),(-?\d+)>')

def parse_input(lines):
    x = []
    v = []
    a = []
    for l in lines:
        m = PARTICLE_RE.match(l)
        x.append([int(x) for x in m.group(1, 2, 3)])
        v.append([int(x) for x in m.group(4, 5, 6)])
        a.append([int(x) for x in m.group(7, 8, 9)])

    return (np.arange(len(x)), np.array(x), np.array(v), np.array(a))

i, x, v, a = parse_input(fi.input())

Now we can calculate which particle will be closest to the origin in the long-term: this is simply the particle with the smallest acceleration. It turns out that several have the same acceleration, so of these, the one we want is the one with the lowest starting velocity. This is only complicated slightly by the need to get the number of the particle rather than its other information, hence the need to use numpy.argmin.

a_abs = np.sum(np.abs(a), axis=1)
a_min = np.min(a_abs)
a_i = np.squeeze(np.argwhere(a_abs == a_min))
closest = i[a_i[np.argmin(np.sum(np.abs(v[a_i]), axis=1))]]
print("Closest: ", closest)

Now we define functions to simulate collisions between particles. We have to use the return_index and return_counts options to numpy.unique to be able to get rid of all the duplicate positions (the standard usage is to keep one of each duplicate).

def resolve_collisions(x, v, a):
    (_, i, c) = np.unique(x, return_index=True, return_counts=True, axis=0)
    i = i[c == 1]
    return x[i], v[i], a[i]

The termination criterion for this loop is an interesting aspect: the most robust to my mind seems to be that eventually the particles will end up sorted in order of their initial acceleration in terms of distance from the origin, so you could check for this but that’s pretty computationally expensive. In the end, all that was needed was a bit of trial and error: terminating arbitrarily after 1,000 iterations seems to work! In fact, all the collisions are over after about 40 iterations for my input but there was always the possibility that two particles with very slightly different accelerations would eventually intersect much later.

def simulate_collisions(x, v, a, iterations=1000):
    for _ in range(iterations):
        v += a
        x += v
        x, v, a = resolve_collisions(x, v, a)

    return len(x)

print("Remaining particles: ", simulate_collisions(x, v, a))

A Series of Tubes — Rust — #adventofcode Day 19

Sun, 24 Dec 2017 18:15:28 +0000

Today’s challenge asks us to help a network packet find its way.

→ Full code on GitHub

!!! commentary Today’s challenge was fairly straightforward, following an ASCII art path, so I thought I’d give Rust another try. I’m a bit behind on the blog posts, so I’m presenting the code below without any further commentary. I’m not really convinced this is good idiomatic Rust, and it was interesting turning a set of strings into a 2D array of characters because there are both u8 (byte) and char types to deal with.

use std::io;
use std::io::BufRead;

const ALPHA: &'static str = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";

fn change_direction(dia: &Vec<Vec<u8>>, x: usize, y: usize, dx: &mut i32, dy: &mut i32) {
    assert_eq!(dia[x][y], b'+');
    
    if dx.abs() == 1 {
        *dx = 0;
        if y + 1 < dia[x].len() && (dia[x][y + 1] == b'-' || ALPHA.contains(dia[x][y + 1] as char)) {
            *dy = 1;
        } else if dia[x][y - 1] == b'-' || ALPHA.contains(dia[x][y - 1] as char) {
            *dy = -1;
        } else {
            panic!("Huh? {} {}", dia[x][y+1] as char, dia[x][y-1] as char);
        }
    } else {
        *dy = 0;
        if x + 1 < dia.len() && (dia[x + 1][y] == b'|' || ALPHA.contains(dia[x + 1][y] as char)) {
            *dx = 1;
        } else if dia[x - 1][y] == b'|' || ALPHA.contains(dia[x - 1][y] as char) {
            *dx = -1;
        } else {
            panic!("Huh?");
        }
    }
}

fn follow_route(dia: Vec<Vec<u8>>) -> (String, i32) {
    let mut x: i32 = 0;
    let mut y: i32;
    let mut dx: i32 = 1;
    let mut dy: i32 = 0;
    let mut result = String::new();
    let mut steps = 1;

    match dia[0].iter().position(|x| *x == b'|') {
        Some(i) => y = i as i32,
        None => panic!("Could not find '|' in first row"),
    }

    loop {
        x += dx;
        y += dy;
        match dia[x as usize][y as usize] {
            b'A'...b'Z' => result.push(dia[x as usize][y as usize] as char),
            b'+' => change_direction(&dia, x as usize, y as usize, &mut dx, &mut dy),
            b' ' => return (result, steps),
            _ => (),
        }
        steps += 1;
    }
}

fn main() {
    let stdin = io::stdin();
    let lines: Vec<Vec<u8>> = stdin.lock().lines()
        .map(|l| l.unwrap().into_bytes())
        .collect();

    let result = follow_route(lines);
    println!("Route: {}", result.0);
    println!("Steps: {}", result.1);
}

Duet — Haskell — #adventofcode Day 18

Sun, 24 Dec 2017 17:59:01 +0000

Today’s challenge introduces a type of simplified assembly language that includes instructions for message-passing. First we have to simulate a single program (after humorously misinterpreting the snd and rcv instructions as “sound” and “recover”), but then we have to simulate two concurrent processes and the message passing between them.

→ Full code on GitHub

!!! commentary Well, I really learned a lot from this one! I wanted to get to grips with more complex stuff in Haskell and this challenge seemed like an excellent opportunity to figure out a) parsing with the parsec library and b) using the State monad to keep the state of the simulator.

As it turned out, that wasn't all I'd learned: I also ran into an interesting situation whereby lazy evaluation was creating an infinite loop where there shouldn't be one, so I also had to learn how to selectively force strict evaluation of values. I'm pretty sure this isn't the best Haskell in the world, but I'm proud of it.

First we have to import a bunch of stuff to use later, but also notice the pragma on the first line which instructs the compiler to enable the BangPatterns language extension, which will be important later.

{-# LANGUAGE BangPatterns #-}
module Main where

import qualified Data.Vector as V
import qualified Data.Map.Strict as M
import Data.List
import Data.Either
import Data.Maybe
import Control.Monad.State.Strict
import Control.Monad.Loops
import Text.ParserCombinators.Parsec hiding (State)

First up we define the types that will represent the program code itself.

data DuetVal = Reg Char | Val Int deriving Show
type DuetQueue = [Int]
data DuetInstruction = Snd DuetVal
                       | Rcv DuetVal
                       | Jgz DuetVal DuetVal
                       | Set DuetVal DuetVal
                       | Add DuetVal DuetVal
                       | Mul DuetVal DuetVal
                       | Mod DuetVal DuetVal
                        deriving Show
type DuetProgram = V.Vector DuetInstruction

Next we define the types to hold the machine state, which includes: registers, instruction pointer, send & receive buffers and the program code, plus a counter of the number of sends made (to provide the solution).

type DuetRegisters = M.Map Char Int
data Duet = Duet { dRegisters :: DuetRegisters
                 , dPtr :: Int
                 , dSendCount :: Int
                 , dRcvBuf :: DuetQueue
                 , dSndBuf :: DuetQueue
                 , dProgram :: DuetProgram }

instance Show Duet where
  show d = show (dRegisters d) ++ " @" ++ show (dPtr d) ++ " S" ++ show (dSndBuf d) ++ " R" ++ show (dRcvBuf d)

defaultDuet = Duet M.empty 0 0 [] [] V.empty

type DuetState = State Duet

program is a parser built on the cool parsec library to turn the program text into a Haskell format that we can work with, a Vector of instructions. Yes, using a full-blown parser is overkill here (it would be much simpler just to split each line on whitespace, but I wanted to see how Parsec works. I’m using Vector here because we need random access to the instruction list, which is much more efficient with Vector: O(1) compared with the O(n) of the built in Haskell list ([]) type. parseProgram applies the parser to a string and returns the result.

program :: GenParser Char st DuetProgram
program = do
  instructions <- endBy instruction eol
  return $ V.fromList instructions
  where
    instruction = try (oneArg "snd" Snd) <|> oneArg "rcv" Rcv
                  <|> twoArg "set" Set <|> twoArg "add" Add
                  <|> try (twoArg "mul" Mul)
                  <|> twoArg "mod" Mod <|> twoArg "jgz" Jgz
    oneArg n c = do
      string n >> spaces
      val <- regOrVal
      return $ c val
    twoArg n c = do
      string n >> spaces
      val1 <- regOrVal
      spaces
      val2 <- regOrVal
      return $ c val1 val2
    regOrVal = register <|> value
    register = do
      name <- lower
      return $ Reg name
    value = do
      val <- many $ oneOf "-0123456789"
      return $ Val $ read val
    eol = char '\n'

parseProgram :: String -> Either ParseError DuetProgram
parseProgram = parse program ""

Next up we have some utility functions that sit in the DuetState monad we defined above and perform common manipulations on the state: getting/setting/updating registers, updating the instruction pointer and sending/receiving messages via the relevant queues.

getReg :: Char -> DuetState Int
getReg r = do
  st <- get
  return $ M.findWithDefault 0 r (dRegisters st)

putReg :: Char -> Int -> DuetState ()
putReg r v = do
  st <- get
  let current = dRegisters st
      new = M.insert r v current
  put $ st { dRegisters = new }

modReg :: (Int -> Int -> Int) -> Char -> DuetVal -> DuetState Bool
modReg op r v = do
  u <- getReg r
  v' <- getRegOrVal v
  putReg r (u `op` v')
  incPtr
  return False

getRegOrVal :: DuetVal -> DuetState Int
getRegOrVal (Reg r) = getReg r
getRegOrVal (Val v) = return v

addPtr :: Int -> DuetState ()
addPtr n = do
  st <- get
  put $ st { dPtr = n + dPtr st }

incPtr = addPtr 1

send :: Int -> DuetState ()
send v = do
  st <- get
  put $ st { dSndBuf = (dSndBuf st ++ [v]), dSendCount = dSendCount st + 1 }

recv :: DuetState (Maybe Int)
recv = do
  st <- get
  case dRcvBuf st of
    (x:xs) -> do
      put $ st { dRcvBuf = xs }
      return $ Just x
    [] -> return Nothing

execInst implements the logic for each instruction. It returns False as long as the program can continue, but True if the program tries to receive from an empty buffer.

execInst :: DuetInstruction -> DuetState Bool
execInst (Set (Reg reg) val) = do
  newVal <- getRegOrVal val
  putReg reg newVal
  incPtr
  return False
execInst (Mul (Reg reg) val) = modReg (*) reg val
execInst (Add (Reg reg) val) = modReg (+) reg val
execInst (Mod (Reg reg) val) = modReg mod reg val
execInst (Jgz val1 val2) = do
  st <- get
  test <- getRegOrVal val1
  jump <- if test > 0 then getRegOrVal val2 else return 1
  addPtr jump
  return False
execInst (Snd val) = do
  v <- getRegOrVal val
  send v
  st <- get
  incPtr
  return False
execInst (Rcv (Reg r)) = do
  st <- get
  v <- recv
  handle v
  where
    handle :: Maybe Int -> DuetState Bool
    handle (Just x) = putReg r x >> incPtr >> return False
    handle Nothing = return True
execInst x = error $ "execInst not implemented yet for " ++ show x

execNext looks up the next instruction and executes it. runUntilWait runs the program until execNext returns True to signal the wait state has been reached.

execNext :: DuetState Bool
execNext = do
  st <- get
  let prog = dProgram st
      p = dPtr st
  if p >= length prog then return True else execInst (prog V.! p)

runUntilWait :: DuetState ()
runUntilWait = do
  waiting <- execNext
  unless waiting runUntilWait

runTwoPrograms handles the concurrent running of two programs, by running first one and then the other to a wait state, then swapping each program’s send buffer to the other’s receive buffer before repeating.

If you look carefully, you’ll see a “bang” (!) before the two arguments of the function: runTwoPrograms !d0 !d1. Haskell is a lazy language and usually doesn’t evaluate a computation until you ask for a result, instead carrying around a “thunk” or plan for how to carry out the computation. Sometimes that can be a problem because the amount of memory your program is using can explode unnecessarily as a long computation turns into a large thunk which isn’t evaluated until the very end. That’s not the problem here though.

What happens here without the bangs is another side-effect of laziness. The exit condition of this recursive function is that a deadlock has been reached: both programs are waiting to receive, but neither has sent anything, so neither can ever continue. The check for this is (null $ dSndBuf d0') && (null $ dSndBuf d1'). As long as the first program has something in its send buffer, the test fails without ever evaluating the second part, which means the result d1' of running the second program is never needed. The function immediately goes to the recursive case and tries to continue the first program again, which immediately returns because it’s still waiting to receive. The same thing happens again, and the result is that instead of running the second program to obtain something for the first to receive, we get into an infinite loop trying and failing to continue the first program.

The bang forces both d0 and d1 to be evaluated at the point we recurse, which forces the rest of the computation: running the second program and swapping the send/receive buffers. With that, the evaluation proceeds correctly and we terminate with a result instead of getting into an infinite loop!

runTwoPrograms :: Duet -> Duet -> (Int, Int)
runTwoPrograms !d0 !d1
  | (null $ dSndBuf d0') && (null $ dSndBuf d1') = (dSendCount d0', dSendCount d1')
  | otherwise = runTwoPrograms d0'' d1''
  where
    (_, d0') = runState runUntilWait d0
    (_, d1') = runState runUntilWait d1
    d0'' = d0' { dSndBuf = [], dRcvBuf = dSndBuf d1' }
    d1'' = d1' { dSndBuf = [], dRcvBuf = dSndBuf d0' }

All that remains to be done now is to run the programs and see how many messages were sent before the deadlock.

main = do
  prog <- fmap (fromRight V.empty . parseProgram) getContents  
  let d0 = defaultDuet { dProgram = prog, dRegisters = M.fromList [('p', 0)] }
      d1 = defaultDuet { dProgram = prog, dRegisters = M.fromList [('p', 1)] }
      (send0, send1) = runTwoPrograms d0 d1
  putStrLn $ "Program 0 sent " ++ show send0 ++ " messages"
  putStrLn $ "Program 1 sent " ++ show send1 ++ " messages"

Spinlock — Rust/Python — #adventofcode Day 17

Sun, 17 Dec 2017 20:03:38 +0000

In today’s challenge we deal with a monstrous whirlwind of a program, eating up CPU and memory in equal measure.

→ Full code on GitHub (and Python driver script)

!!! commentary One of the things I wanted from AoC was an opportunity to try out some popular languages that I don’t currently know, including the memory-safe, strongly-typed compiled languages Go and Rust. Realistically though, I’m likely to continue doing most of my programming in Python, and use one of these other languages when it has better tools or I need the extra speed.

In which case, what I really want to know is how I can call functions written in Go or Rust from Python. I thought I'd try Rust first, as it seems to be designed to be C-compatible and that makes it easy to call from Python using [`ctypes`](https://docs.python.org/3.6/library/ctypes.html).

Part 1 was another straightforward simulation: translate what the "spinlock" monster is doing into code and run it. It was pretty obvious from the story of this challenge and experience of the last few days that this was going to be another one where the simulation is too computationally expensive for part two, which turns out to be correct.

So, first thing to do is to implement the meat of the solution in Rust. spinlock solves the first part of the problem by doing exactly what the monster does. Since we only have to go up to 2017 iterations, this is very tractable. The last number we insert is 2017, so we just return the number immediately after that.

#[no_mangle]
pub extern fn spinlock(n: usize, skip: usize) -> i32 {
    let mut buffer: Vec<i32> = Vec::with_capacity(n+1);
    buffer.push(0);
    buffer.push(1);
    let mut pos = 1;

    for i in 2..n+1 {
        pos = (pos + skip + 1) % buffer.len();
        buffer.insert(pos, i as i32);
    }

    pos = (pos + 1) % buffer.len();

    return buffer[pos];
}

For the second part, we have to do 50 million iterations instead, which is a lot. Given that every time you insert an item in the list it has to move up all the elements after that position, I’m pretty sure the algorithm is O(n^2), so it’s going to take a lot longer than 10,000ish times the first part.

Thankfully, we don’t need to build the whole list, just keep track of where 0 is and what number is immediately after it. There may be a closed-form solution to simply calculate the result, but I couldn’t think of it and this is good enough.

#[no_mangle]
pub extern fn spinlock0(n: usize, skip: usize) -> i32 {
    let mut pos = 1;
    let mut pos_0 = 0;
    let mut after_0 = 1;

    for i in 2..n+1 {
        pos = (pos + skip + 1) % i;
        if pos == pos_0 + 1 {
            after_0 = i;
        }
        if pos <= pos_0 {
            pos_0 += 1;
        }
    }

    return after_0 as i31;
}

Now it’s time to call this code from Python. Notice the #[no_mangle] pragmas and pub extern declarations for each function above, which are required to make sure the functions are exported in a C-compatible way. We can build this into a shared library like this:

rustc --crate-type=cdylib -o spinlock.so 17-spinlock.rs

The Python script is as simple as loading this library, reading the puzzle input from the command line and calling the functions. The ctypes module does a lot of magic so that we don’t have to worry about converting from Python types to native types and back again.

import ctypes
import sys

lib = ctypes.cdll.LoadLibrary("./spinlock.so")

skip = int(sys.argv[1])
print("Part 1:", lib.spinlock(2017, skip))
print("Part 2:", lib.spinlock0(50_000_000, skip))

This is a toy example as far as calling Rust from Python is concerned, but it’s worth noting that already we can play with the parameters to the two Rust functions without having to recompile. For more serious work, I’d probably be looking at something like PyO3 to make a proper Python module. Looks like there’s also a very early Rust numpy integration for integrating numerical stuff.

You can also do the same thing from Julia, which has a ccall function built in:

ccall((:spinlock, "./spinlock.so"), Int32, (UInt64, UInt64), 2017, 377)

My next thing to try might be Haskell → Python though…

Permutation Promenade — Julia — #adventofcode Day 16

Sat, 16 Dec 2017 15:58:38 +0000

Today’s challenge rather appeals to me as a folk dancer, because it describes a set of instructions for a dance and asks us to work out the positions of the dancing programs after each run through the dance.

→ Full code on GitHub

!!! commentary So, part 1 is pretty straight forward: parse the set of instructions, interpret them and keep track of the dancer positions as you go. One time through the dance. However, part 2 asks for the positions after 1 billion (yes, that’s 1,000,000,000) times through the dance. In hindsight I should have immediately become suspicious, but I thought I’d at least try the brute force approach first because it was simpler to code.

So I give it a try, and after waiting for a while, having a cup of tea etc. it still hasn't terminated. I try reducing the number of iterations to 1,000. Now it terminates, but takes about 6 seconds. A spot of arithmetic suggests that running the full version will take a little over 190 years. There must be a better way than that!

I'm a little embarassed that I didn't spot the solution immediately (blaming Julia) and tried again in Python to see if I could get it to terminate quicker. When that didn't work I had to think again. A little further investigation with a while loop shows that in fact the dance position repeats (in the case of my input) every 48 times. After that it becomes much quicker!

Oh, and it was time for a new language, so I wasted some extra time working out the quirks of [Julia][].

First, a function to evaluate a single move — for neatness, this dispatches to a dedicated function depending on the type of move, although this isn’t really necessary to solve the challenge. Ending a function name with a bang (!) is a Julia convention to indicate that it has side-effects.

function eval_move!(move, dancers)
    move_type = move[1]
    params = move[2:end]
    if move_type == 's' # spin
        eval_spin!(params, dancers)
    elseif move_type == 'x' # exchange
        eval_exchange!(params, dancers)
    elseif move_type == 'p' # partner swap
        eval_partner!(params, dancers)
    end
end

These take care of the individual moves. Parsing the parameters from a string every single time probably isn’t ideal, but as it turns out, that optimisation isn’t really necessary. Note the + 1 in eval_exchange!, which is necessary because Julia is one of those crazy languages where indexes start from 1 instead of 0. These actions are pretty nice to implement, because Julia has circshift as a builtin to rotate a list, and allows you to assign to list slices and swap values in place with a single statement.

function eval_spin!(params, dancers)
    shift = parse(Int, params)
    dancers[1:end] = circshift(dancers, shift)
end

function eval_exchange!(params, dancers)
    i, j = map(x -> parse(Int, x) + 1, split(params, "/"))
    dancers[i], dancers[j] = dancers[j], dancers[i]
end

function eval_partner!(params, dancers)
    a, b = split(params, "/")
    ia = findfirst([x == a for x in dancers])
    ib = findfirst([x == b for x in dancers])
    dancers[ia], dancers[ib] = b, a
end

dance! takes a list of moves and takes the dances once through the dance.

function dance!(moves, dancers)
    for m in moves
        eval_move!(m, dancers)
    end
end

To solve part 1, we simply need to read the moves in, set up the initial positions of the dances and run the dance through once. join is necessary to a) turn characters into length-1 strings, and b) convert the list of strings back into a single string to print out.

moves = split(readchomp(STDIN), ",")
dancers = collect(join(c) for c in 'a':'p')
orig_dancers = copy(dancers)

dance!(moves, dancers)
println(join(dancers))

Part 2 requires a little more work. We run the dance through again and again until we get back to the initial position, saving the intermediate positions in a list. The list now contains every possible position available from that starting point, so we can find position 1 billion by taking 1,000,000,000 modulo the list length (plus 1 because 1-based indexing) and use that to index into the list to get the final position.

dance_cycle = [orig_dancers]
while dancers != orig_dancers
    push!(dance_cycle, copy(dancers))
    dance!(moves, dancers)
end

println(join(dance_cycle[1_000_000_000 % length(dance_cycle) + 1]))

This terminates on my laptop in about 1.6s: Brute force 0; Careful thought 1!

Dueling Generators — Rust — #adventofcode Day 15

Fri, 15 Dec 2017 18:24:09 +0000

Today’s challenge introduces two pseudo-random number generators which are trying to agree on a series of numbers. We play the part of the “judge”, counting the number of times their numbers agree in the lowest 16 bits.

→ Full code on GitHub

Ever since I used Go to solve day 3, I’ve had a hankering to try the other new kid on the memory-safe compiled language block, Rust. I found it a bit intimidating at first because the syntax wasn’t as close to the C/C++ I’m familiar with and there are quite a few concepts unique to Rust, like the use of traits. But I figured it out, so I can tick another language of my to-try list.

I also implemented a version in Python for comparison: the Python version is more concise and easier to read but the Rust version runs about 10× faster.

First we include the std::env “crate” which will let us get access to commandline arguments, and define some useful constants for later.

use std::env;

const M: i64 = 2147483647;
const MASK: i64 = 0b1111111111111111;
const FACTOR_A: i64 = 16807;
const FACTOR_B: i64 = 48271;

gen_next generates the next number for a given generator’s sequence. gen_next_picky does the same, but for the “picky” generators, only returning values that meet their criteria.

fn gen_next(factor: i64, current: i64) -> i64 {
    return (current * factor) % M;
}

fn gen_next_picky(factor: i64, current: i64, mult: i64) -> i64 {
    let mut next = gen_next(factor, current);
    while next % mult != 0 {
        next = gen_next(factor, next);
    }
    return next;
}

duel runs a single duel, and returns the number of times the generators agreed in the lowest 16 bits (found by doing a binary & with the mask defined above). Rust allows functions to be passed as parameters, so we use this to be able to run both versions of the duel using only this one function.

fn duel<F, G>(n: i64, next_a: F, mut value_a: i64, next_b: G, mut value_b: i64) -> i64
where
    F: Fn(i64) -> i64,
    G: Fn(i64) -> i64,
{
    let mut count = 0;

    for _ in 0..n {
        value_a = next_a(value_a);
        value_b = next_b(value_b);
        if (value_a & MASK) == (value_b & MASK) {
            count += 1;
        }
    }

    return count;
}

Finally, we read the start values from the command line and run the two duels. The expressions that begin |n| are closures (anonymous functions, often called lambdas in other languages) that we use to specify the generator functions for each duel.

fn main() {
    let args: Vec<String> = env::args().collect();

    let start_a: i64 = args[1].parse().unwrap();
    let start_b: i64 = args[2].parse().unwrap();

    println!(
        "Duel 1: {}",
        duel(
            40000000,
            |n| gen_next(FACTOR_A, n),
            start_a,
            |n| gen_next(FACTOR_B, n),
            start_b,
        )
    );
    println!(
        "Duel 2: {}",
        duel(
            5000000,
            |n| gen_next_picky(FACTOR_A, n, 4),
            start_a,
            |n| gen_next_picky(FACTOR_B, n, 8),
            start_b,
        )
    );
}

Disk Defragmentation — Haskell — #adventofcode Day 14

Fri, 15 Dec 2017 17:52:46 +0000

Today’s challenge has us helping a disk defragmentation program by identifying contiguous regions of used sectors on a 2D disk.

→ Full code on GitHub

!!! commentary Wow, today’s challenge had a pretty steep learning curve. Day 14 was the first to directly reuse code from a previous day: the “knot hash” from day 10. I solved day 10 in Haskell, so I thought it would be easier to stick with Haskell for today as well. The first part was straightforward, but the second was pretty mind-bending in a pure functional language!

I ended up solving it by implementing a [flood fill algorithm][flood]. It's recursive, which is right in Haskell's wheelhouse, but I ended up using `Data.Sequence` instead of the standard list type as its API for indexing is better. I haven't tried it, but I think it will also be a little faster than a naive list-based version.

It took a looong time to figure everything out, but I had a day off work to be able to concentrate on it!

A lot more imports for this solution, as we’re exercising a lot more of the standard library.

module Main where

import Prelude hiding (length, filter, take)
import Data.Char (ord)
import Data.Sequence
import Data.Foldable hiding (length)
import Data.Ix (inRange)
import Data.Function ((&))
import Data.Maybe (fromJust, mapMaybe, isJust)
import qualified Data.Set as Set
import Text.Printf (printf)
import System.Environment (getArgs)

Also we’ll extract the key bits from day 10 into a module and import that.

import KnotHash

Now we define a few data types to make the code a bit more readable. Sector represent the state of a particular disk sector, either free, used (but unmarked) or used and marked as belonging to a given integer-labelled group. Grid is a 2D matrix of Sector, as a sequence of sequences.

data Sector = Free | Used | Mark Int
  deriving (Eq)

instance Show Sector where
  show Free = "   ."
  show Used = "   #"
  show (Mark i) = printf "%4d" i

type GridRow = Seq Sector
type Grid = Seq (GridRow)

Some utility functions to make it easier to view the grids (which can be quite large): used for debugging but not in the finished solution.

subGrid :: Int -> Grid -> Grid
subGrid n = fmap (take n) . take n

printRow :: GridRow -> IO ()
printRow row = do
  mapM_ (putStr . show) row
  putStr "\n"

printGrid :: Grid -> IO ()
printGrid = mapM_ printRow

makeKey generates the hash key for a given row.

makeKey :: String -> Int -> String
makeKey input n = input ++ "-" ++ show n

stringToGridRow converts a binary string of ‘1’ and ‘0’ characters to a sequence of Sector values.

stringToGridRow :: String -> GridRow
stringToGridRow = fromList . map convert
  where convert x
          | x == '1' = Used
          | x == '0' = Free

makeRow and makeGrid build up the grid to use based on the provided input string.

makeRow :: String -> Int -> GridRow
makeRow input n = stringToGridRow $ concatMap (printf "%08b")
  $ dense $ fullKnotHash 256
  $ map ord $ makeKey input n

makeGrid :: String -> Grid
makeGrid input = fromList $ map (makeRow input) [0..127]

Utility functions to count the number of used and free sectors, to give the solution to part 1.

countEqual :: Sector -> Grid -> Int
countEqual x = sum . fmap (length . filter (==x))

countUsed = countEqual Used
countFree = countEqual Free

Now the real meat begins! fundUnmarked finds the location of the next used sector that we haven’t yet marked. It returns a Maybe value, which is Just (x, y) if there is still an unmarked block or Nothing if there’s nothing left to mark.

findUnmarked :: Grid -> Maybe (Int, Int)
findUnmarked g
  | y == Nothing = Nothing
  | otherwise = Just (fromJust x, fromJust y)
  where
    hasUnmarked row = isJust $ elemIndexL Used row
    x = findIndexL hasUnmarked g
    y = case x of
      Nothing -> Nothing
      Just x' -> elemIndexL Used $ index g x'

floodFill implements a very simple recursive flood fill. It takes a target and replacement value and a starting location, and fills in the replacement value for every connected location that currently has the target value. We use it below to replace a connected used region with a marked region.

floodFill :: Sector -> Sector -> (Int, Int) -> Grid -> Grid
floodFill t r (x, y) g
  | inRange (0, length g - 1) x
    && inRange (0, length g - 1) y
    && elem == t =
      let newRow = update y r row
          newGrid = update x newRow g
      in newGrid
         & floodFill t r (x+1, y)
         & floodFill t r (x-1, y)
         & floodFill t r (x, y+1)
         & floodFill t r (x, y-1)
  | otherwise = g
  where
    row = g `index` x
    elem = row `index` y

markNextGroup looks for an unmarked group and marks it if found. If no more groups are found it returns Nothing. markAllGroups then repeatedly applies markNextGroup until Nothing is returned.

markNextGroup :: Int -> Grid -> Maybe Grid
markNextGroup i g = case findUnmarked g of
                      Nothing -> Nothing
                      Just loc -> Just $ floodFill Used (Mark i) loc g

markAllGroups :: Grid -> Grid
markAllGroups g = markAllGroups' 1 g
  where
    markAllGroups' i g = case markNextGroup i g of
      Nothing -> g
      Just g' -> markAllGroups' (i+1) g'

onlyMarks filters a grid row and returns a list of (possibly duplicated) group numbers in the row.

onlyMarks :: GridRow -> [Int]
onlyMarks = mapMaybe getMark . toList
  where
    getMark Free = Nothing
    getMark Used = Nothing
    getMark (Mark i) = Just i

Finally, countGroups puts all the group numbers into a set to get rid of duplicates and returns the size of the set, i.e. the total number of separate groups.

countGroups :: Grid -> Int
countGroups g = Set.size groupSet
  where
    groupSet = foldl' Set.union Set.empty $ fmap rowToSet g
    rowToSet = Set.fromList . toList . onlyMarks

As always, every Haskell program needs a main function to drive the I/O and produce the actual result.

main = do
  input <- fmap head getArgs
  let grid = makeGrid input
      used = countUsed grid
      marked = countGroups $ markAllGroups grid

  putStrLn $ "Used sectors: " ++ show used
  putStrLn $ "Groups: " ++ show marked

Packet Scanners — Haskell — #adventofcode Day 13

Fri, 15 Dec 2017 15:56:20 +0000

Today’s challenge requires us to sneak past a firewall made up of a series of scanners.

→ Full code on GitHub

!!! commentary I wasn’t really thinking straight when I solved this challenge. I got a solution without too much trouble, but I ended up simulating the step-by-step movement of the scanners.

I finally realised that I could calculate whether or not a given scanner was safe at a given time directly with modular arithmetic, and it bugged me so much that I reimplemented the solution. Both are given below, the faster one first.

First we introduce some standard library stuff and define some useful utilities.

module Main where

import qualified Data.Text as T
import Data.Maybe (mapMaybe)

strip :: String -> String
strip = T.unpack . T.strip . T.pack

splitOn :: String -> String -> [String]
splitOn sep = map T.unpack . T.splitOn (T.pack sep) . T.pack

parseScanner :: String -> (Int, Int)
parseScanner s = (d, r)
  where [d, r] = map read $ splitOn ": " s

traverseFW does all the hard work: it checks for each scanner whether or not it’s safe as we pass through, and returns a list of the severities of each time we’re caught. mapMaybe is like the standard map in many languages, but operates on a list of Haskell Maybe values, like a combined map and filter. If the value is Just x, x gets included in the returned list; if the value is Nothing, then it gets thrown away.

traverseFW :: Int -> [(Int, Int)] -> [Int]
traverseFW delay = mapMaybe caught
  where
    caught (d, r) = if (d + delay) `mod` (2*(r-1)) == 0
      then Just (d * r)
      else Nothing

Then the total severity of our passage through the firewall is simply the sum of each individual severity.

severity :: [(Int, Int)] -> Int
severity = sum . traverseFW 0

But we don’t want to know how badly we got caught, we want to know how long to wait before setting off to get through safely. findDelay tries traversing the firewall with increasing delay, and returns the delay for the first pass where we predict not getting caught.

findDelay :: [(Int, Int)] -> Int
findDelay scanners = head $ filter (null . flip traverseFW scanners) [0..]

And finally, we put it all together and calculate and print the result.

main = do
  scanners <- fmap (map parseScanner . lines) getContents

  putStrLn $ "Severity: " ++ (show $ severity scanners)
  putStrLn $ "Delay: " ++ (show $ findDelay scanners)

I’m not generally bothered about performance for these challenges, but here I’ll note that my second attempt runs in a little under 2 seconds on my laptop:

$ time ./13-packet-scanners-redux < 13-input.txt
Severity: 1900
Delay: 3966414
./13-packet-scanners-redux < 13-input.txt  1.73s user 0.02s system 99% cpu 1.754 total

Compare that with the first, simulation-based one, which takes nearly a full minute:

$ time ./13-packet-scanners < 13-input.txt
Severity: 1900
Delay: 3966414
./13-packet-scanners < 13-input.txt  57.63s user 0.27s system 100% cpu 57.902 total

And for good measure, here’s the code. Notice the tick and tickOne functions, which together simulate moving all the scanners by one step; for this to work we have to track the full current state of each scanner, which is easier to read with a Haskell record-based custom data type. traverseFW is more complicated because it has to drive the simulation, but the rest of the code is mostly the same.

module Main where

import qualified Data.Text as T
import Control.Monad (forM_)

data Scanner = Scanner { depth :: Int
                       , range :: Int
                       , pos :: Int
                       , dir :: Int }
instance Show Scanner where
  show (Scanner d r p dir) = show d ++ "/" ++ show r ++ "/" ++ show p ++ "/" ++ show dir

strip :: String -> String
strip = T.unpack . T.strip . T.pack

splitOn :: String -> String -> [String]
splitOn sep str = map T.unpack $ T.splitOn (T.pack sep) $ T.pack str

parseScanner :: String -> Scanner
parseScanner s = Scanner d r 0 1
  where [d, r] = map read $ splitOn ": " s

tickOne :: Scanner -> Scanner
tickOne (Scanner depth range pos dir)
  | pos <= 0         = Scanner depth range (pos+1) 1
  | pos >= range - 1 = Scanner depth range (pos-1) (-1)
  | otherwise        = Scanner depth range (pos+dir) dir

tick :: [Scanner] -> [Scanner]
tick = map tickOne

traverseFW :: [Scanner] -> [(Int, Int)]
traverseFW = traverseFW' 0
  where
    traverseFW' _ [] = []
    traverseFW' layer scanners@((Scanner depth range pos _):rest)
      -- | layer == depth && pos == 0  = (depth*range) + (traverseFW' (layer+1) $ tick rest)
      | layer == depth && pos == 0  = (depth,range) : (traverseFW' (layer+1) $ tick rest)
      | layer == depth && pos /= 0  = traverseFW' (layer+1) $ tick rest
      | otherwise                   = traverseFW' (layer+1) $ tick scanners

severity :: [Scanner] -> Int
severity = sum . map (uncurry (*)) . traverseFW

empty :: [a] -> Bool
empty [] = True
empty _ = False
  
findDelay :: [Scanner] -> Int
findDelay scanners = delay
  where
    (delay, _) = head $ filter (empty . traverseFW . snd) $ zip [0..] $ iterate tick scanners
    

main = do
  scanners <- fmap (map parseScanner . lines) getContents

  putStrLn $ "Severity: " ++ (show $ severity scanners)
  putStrLn $ "Delay: " ++ (show $ findDelay scanners)

Digital Plumber — Python — #adventofcode Day 12

Tue, 12 Dec 2017 17:43:44 +0000

Today’s challenge has us helping a village of programs who are unable to communicate. We have a list of the the communication channels between their houses, and need to sort them out into groups such that we know that each program can communicate with others in its own group but not any others. Then we have to calculate the size of the group containing program 0 and the total number of groups.

→ Full code on GitHub

!!! commentary This is one of those problems where I’m pretty sure that my algorithm isn’t close to being the most efficient, but it definitely works! For the sake of solving the challenge that’s all that matters, but it still bugs me.

By now I’ve become used to using fileinput to transparently read data either from files given on the command-line or standard input if no arguments are given.

import fileinput as fi

First we make an initial pass through the input data, creating a group for each line representing the programs on that line (which can communicate with each other). We store this as a Python set.

groups = []
for line in fi.input():
    head, rest = line.split(' <-> ')
    group = set([int(head)])
    group.update([int(x) for x in rest.split(', ')])
    groups.append(group)

Now we iterate through the groups, starting with the first, and merging any we find that overlap with our current group.

i = 0
while i < len(groups):
    current = groups[i]

Each pass through the groups brings more programs into the current group, so we have to go through and check their connections too. We make several merge passes, until we detect that no more merges took place.

    num_groups = len(groups) + 1
    while num_groups > len(groups):
        j = i+1
        num_groups = len(groups)

This inner loop does the actual merging, and deletes each group as it’s merged in.

        while j < len(groups):
            if len(current & groups[j]) > 0:
                current.update(groups[j])
                del groups[j]
            else:
                j += 1
    i += 1

All that’s left to do now is to display the results.

print("Number in group 0:", len([g for g in groups if 0 in g][0]))
print("Number of groups:", len(groups))

Hex Ed — Python — #adventofcode Day 11

Mon, 11 Dec 2017 17:15:04 +0000

Today’s challenge is to help a program find its child process, which has become lost on a hexagonal grid. We need to follow the path taken by the child (given as input) and calculate the distance it is from home along with the furthest distance it has been at any point along the path.

→ Full code on GitHub

!!! commentary I found this one quite interesting in that it was very quick to solve. In fact, I got lucky and my first quick implementation (max(abs(l)) below) gave the correct answer in spite of missing an obvious not-so-edge case. Thinking about it, there’s only a ⅓ chance that the first incorrect implementation would give the wrong answer!

The code is shorter, so you get more words today. ☺

There are a number of different co-ordinate systems on a hexagonal grid (I discovered while reading up after solving it…). I intuitively went for the system known as ‘axial’ coordinates, where you pick two directions aligned to the grid as your x and y axes: note that these won’t be perpendicular. I chose ne/sw as the x axis and se/nw as y, but there are three other possible choices. That leads to the following definition for the directions, encoded as numpy arrays because that makes some of the code below neater.

import numpy as np

STEPS = {d: np.array(v) for d, v in
         [('ne', (1, 0)), ('se', (0, -1)), ('s', (-1, -1)),
          ('sw', (-1, 0)), ('nw', (0, 1)), ('n', (1, 1))]}

hex_grid_dist, given a location l calculates the number of steps needed to reach that location from the centre at (0, 0). Notice that we can’t simply use the Manhattan distance here because, for example, one step north takes us to (1, 1), which would give a Manhattan distance of 2. Instead, we can see that moving in the n/s direction allows us to increment or decrement both coordinates at the same time:

If the coordinates have the same sign: move n/s until one of them is zero, then move along the relevant ne or se axis back to the origin; in this case the number of steps is greatest of the absolute values of the two coordinates
If the coordinates have opposite signs: move independently along the ne and se axes to reduce each to 0; this time the number of steps is the sum of the absolute values of the two coordinates

def hex_grid_distance(l):
    if sum(np.sign(l)) == 0:  # i.e. opposite signs
        return sum(abs(l))
    else:
        return max(abs(l))

Now we can read in the path followed by the child and follow it ourselves, tracking the maximum distance from home along the way.

path = input().strip().split(',')

location = np.array((0, 0))
max_distance = 0

for step in map(STEPS.get, path):
    location += step
    max_distance = max(max_distance, hex_grid_distance(location))

distance = hex_grid_distance(location)

print("Child process is at", location, "which is", distance, "steps away")
print("Greatest distance was", max_distance)

Knot Hash — Haskell — #adventofcode Day 10

Sun, 10 Dec 2017 21:18:36 +0000

Today’s challenge asks us to help a group of programs implement a (highly questionable) hashing algorithm that involves repeatedly reversing parts of a list of numbers.

→ Full code on GitHub

!!! commentary I went with Haskell again today, because it’s the weekend so I have a bit more time, and I really enjoyed yesterday’s Haskell implementation. Today gave me the opportunity to explore the standard library a bit more, as well as lending itself nicely to being decomposed into smaller parts to be combined using higher-order functions.

You know the drill by know: import stuff we’ll use later.

module Main where

import Data.Char (ord)
import Data.Bits (xor)
import Data.Function ((&))
import Data.List (unfoldr)
import Text.Printf (printf)
import qualified Data.Text as T

The worked example uses a concept of the “current position” as a pointer to a location in a static list. In Haskell it makes more sense to instead use the front of the list as the current position, and rotate the whole list as we progress to bring the right element to the front.

rotate :: Int -> [Int] -> [Int]
rotate 0 xs = xs
rotate n xs = drop n' xs ++ take n' xs
  where n' = n `mod` length xs

The simple version of the hash requires working through the input list, modifying the working list as we go, and incrementing a “skip” counter with each step. Converting this to a functional style, we simply zip up the input with an infinite list [0, 1, 2, 3, ...] to give the counter values. Notice that we also have to calculate how far to rotate the working list to get back to its original position. foldl lets us specify a function that returns a modified version of the working list and feeds the input list in one at a time.

simpleKnotHash :: Int -> [Int] -> [Int]
simpleKnotHash size input = foldl step [0..size-1] input' & rotate (negate finalPos)
  where
    input' = zip input [0..]
    finalPos = sum $ zipWith (+) input [0..]
    reversePart xs n  = (reverse $ take n xs) ++ drop n xs
    step xs (n, skip) = reversePart xs n & rotate (n+skip)

The full version of the hash (part 2 of the challenge) starts the same way as the simple version, except making 64 passes instead of one: we can do this by using replicate to make a list of 64 copies, then collapse that into a single list with concat.

fullKnotHash :: Int -> [Int] -> [Int]
fullKnotHash size input = simpleKnotHash size input'
  where input' = concat $ replicate 64 input

The next step in calculating the full hash collapses the full 256-element “sparse” hash down into 16 elements by XORing groups of 16 together. unfoldr is a nice efficient way of doing this.

dense :: [Int] -> [Int]
dense = unfoldr dense'
  where
    dense' [] = Nothing
    dense' xs = Just (foldl1 xor $ take 16 xs, drop 16 xs)

The final hash step is to convert the list of integers into a hexadecimal string.

hexify :: [Int] -> String
hexify = concatMap (printf "%02x")

These two utility functions put together building blocks from the Data.Text module to parse the input string. Note that no arguments are given: the functions are defined purely by composing other functions using the . operator. In Haskell this is referred to as “point-free” style.

strip :: String -> String
strip = T.unpack . T.strip . T.pack

parseInput :: String -> [Int]
parseInput = map (read . T.unpack) . T.splitOn (T.singleton ',') . T.pack

Now we can put it all together, including building the weird input for the “full” hash.

main = do
  input <- fmap strip getContents
  let simpleInput = parseInput input
      asciiInput = map ord input ++ [17, 31, 73, 47, 23]
      (a:b:_) = simpleKnotHash 256 simpleInput
  print $ (a*b)
  putStrLn $ fullKnotHash 256 asciiInput & dense & hexify

Stream Processing — Haskell — #adventofcode Day 9

Sun, 10 Dec 2017 20:57:21 +0000

In today’s challenge we come across a stream that we need to cross. But of course, because we’re stuck inside a computer, it’s not water but data flowing past. The stream is too dangerous to cross until we’ve removed all the garbage, and to prove we can do that we have to calculate a score for the valid data “groups” and the number of garbage characters to remove.

→ Full code on GitHub

!!! commentary One of my goals for this process was to knock the rust of my functional programming skills in Haskell, and I haven’t done that for the whole of the first week. Processing strings character by character and acting according to which character shows up seems like a good choice for pattern-matching though, so here we go. I also wanted to take a bash at test-driven development in Haskell, so I also loaded up the Test.Hspec module to give it a try.

I did find keeping track of all the state in arguments a bit mind boggling, and I think it could have been improved through use of a data type using record syntax and the `State` monad, so that's something to look at for a future challenge.

First import the extra bits we’ll need.

module Main where

import Test.Hspec
import Data.Function ((&))

countGroups solves the first part of the problem, counting up the “score” of the valid data in the stream. countGroups' is an auxiliary function that holds some state in its arguments. We use pattern matching for the base case: [] represents the empty list in Haskell, which indicates we’ve finished the whole stream. Otherwise, we split the remaining stream into its first character and remainder, and use guards to decide how to interpret it. If skip is true, discard the character and carry on with skip set back to false. If we find a “!”, that tells us to skip the next. Other characters mark groups or sets of garbage: groups increase the score when they close and garbage is discarded. We continue to progress the list by recursing with the remainder of the stream and any updated state.

countGroups :: String -> Int
countGroups = countGroups' 0 0 False False
  where
    countGroups' score _ _ _ [] = score
    countGroups' score level garbage skip (c:rest)
      | skip      = countGroups' score level garbage False rest
      | c == '!'  = countGroups' score level garbage True rest
      | garbage   = case c of
                      '>' -> countGroups' score level False False rest
                      _   -> countGroups' score level True False rest
      | otherwise = case c of
                      '{' -> countGroups' score (level+1) False False rest
                      '}' -> countGroups' (score+level) (level-1) False False rest
                      ',' -> countGroups' score level False False rest
                      '<' -> countGroups' score level True False rest
                      c   -> error $ "Garbage character found outside garbage: " ++ show c

countGarbage works almost identically to countGroups, except it ignores groups and counts garbage. They are structured so similarly that it would probably make more sense to combine them to a single function that returns both counts.

countGarbage :: String -> Int
countGarbage = countGarbage' 0 False False
  where
    countGarbage' count _ _ [] = count
    countGarbage' count garbage skip (c:rest)
      | skip      = countGarbage' count garbage False rest
      | c == '!'  = countGarbage' count garbage True rest
      | garbage   = case c of
                      '>' -> countGarbage' count False False rest
                      _   -> countGarbage' (count+1) True False rest
      | otherwise = case c of
                      '<' -> countGarbage' count True False rest
                      _   -> countGarbage' count False False rest

Hspec gives us a domain-specific language heavily inspired by the rspec library for Ruby: the tests read almost like natural language. I built up these tests one-by-one, gradually implementing the appropriate bits of the functions above, a process known as Test-driven development.

runTests = 
  hspec $ do
    describe "countGroups" $ do
      it "counts valid groups" $ do
        countGroups "{}" `shouldBe` 1
        countGroups "{{{}}}" `shouldBe` 6
        countGroups "{{{},{},{{}}}}" `shouldBe` 16
        countGroups "{{},{}}" `shouldBe` 5

      it "ignores garbage" $ do
        countGroups "{<a>,<a>,<a>,<a>}" `shouldBe` 1
        countGroups "{{<ab>},{<ab>},{<ab>},{<ab>}}" `shouldBe` 9

      it "skips marked characters" $ do
        countGroups "{{<!!>},{<!!>},{<!!>},{<!!>}}" `shouldBe` 9
        countGroups "{{<a!>},{<a!>},{<a!>},{<ab>}}" `shouldBe` 3
      

    describe "countGarbage" $ do
      it "counts garbage characters" $ do
        countGarbage "<>" `shouldBe` 0
        countGarbage "<random characters>" `shouldBe` 17
        countGarbage "<<<<>" `shouldBe` 3

      it "ignores non-garbage" $ do
        countGarbage "{{},{}}" `shouldBe` 0
        countGarbage "{{<ab>},{<ab>},{<ab>},{<ab>}}" `shouldBe` 8
        
      it "skips marked characters" $ do
        countGarbage "<{!>}>" `shouldBe` 2
        countGarbage "<!!>" `shouldBe` 0
        countGarbage "<!!!>" `shouldBe` 0
        countGarbage "<{o\"i!a,<{i<a>" `shouldBe` 10

Finally, the main function reads in the challenge input and calculates the answers, printing them on standard output.

main = do
  runTests

  repeat '=' & take 78 & putStrLn

  input <- getContents & fmap (filter (/='\n'))
  putStrLn $ "Found " ++ show (countGroups input) ++ " groups"
  putStrLn $ "Found " ++ show (countGarbage input) ++ " characters garbage"

I Heard You Like Registers — Python — #adventofcode Day 8

Sun, 10 Dec 2017 19:33:46 +0000

Today’s challenge describes a simple instruction set for a CPU, incrementing and decrementing values in registers according to simple conditions. We have to interpret a stream of these instructions, and to prove that we’ve done so, give the highest value of any register, both at the end of the program and throughout the whole program.

→ Full code on GitHub

!!! commentary This turned out to be a nice straightforward one to implement, as the instruction format was easily parsed by regular expression, and Python provides the eval function which made evaluating the conditions a doddle.

Import various standard library bits that we’ll use later.

import re
import fileinput as fi
from math import inf
from collections import defaultdict

We could just parse the instructions by splitting the string, but using a regular expression is a little bit more robust because it won’t match at all if given an invalid instruction.

INSTRUCTION_RE = re.compile(r'(\w+) (inc|dec) (-?\d+) if (.+)\s*')

def parse_instruction(instruction):
    match = INSTRUCTION_RE.match(instruction)
    return match.group(1, 2, 3, 4)

Executing an instruction simply checks the condition and if it evaluates to True updates the relevant register.

def exec_instruction(registers, instruction):
    name, op, value, cond = instruction

    value = int(value)
    if op == 'dec':
        value = -value

    if eval(cond, globals(), registers):
        registers[name] += value

highest_value returns the maximum value found in any register.

def highest_value(registers):
    return sorted(registers.items(), key=lambda x: x[1], reverse=True)[0][1]

Finally, loop through all the instructions and carry them out, updating global_max as we go. We need to be able to deal with registers that haven’t been accessed before. Keeping the registers in a dictionary means that we can evaluate the conditions directly using eval above, passing it as the locals argument. The standard dict will raise an exception if we try to access a key that doesn’t exist, so instead we use collections.defaultdict, which allows us to specify what the default value for a non-existent key will be. New registers start at 0, so we use a simple lambda to define a function that always returns 0.

global_max = -inf
registers = defaultdict(lambda: 0)
for i in map(parse_instruction, fi.input()):
    exec_instruction(registers, i)
    global_max = max(global_max, highest_value(registers))

print('Max value:', highest_value(registers))
print('All-time max:', global_max)

Recursive Circus — Ruby — #adventofcode Day 7

Sun, 10 Dec 2017 19:04:36 +0000

Today’s challenge introduces a set of processes balancing precariously on top of each other. We find them stuck and unable to get down because one of the processes is the wrong size, unbalancing the whole circus. Our job is to figure out the root from the input and then find the correct weight for the single incorrect process.

→ Full code on GitHub

!!! commentary So I didn’t really intend to take a full polyglot approach to Advent of Code, but it turns out to have been quite fun, so I made a shortlist of languages to try. Building a tree is a classic application for object-orientation using a class to represent tree nodes, and I’ve always liked the feel of Ruby’s class syntax, so I gave it a go.

First make sure we have access to Set, which we’ll use later.

require 'set'

Now to define the CircusNode class, which represents nodes in the tree. attr :s automatically creates a function s that returns the value of the instance attribute @s

class CircusNode
  attr :name, :weight

  def initialize(name, weight, children=nil)
    @name = name
    @weight = weight
    @children = children || []
  end

Add a << operator (the same syntax for adding items to a list) that adds a child to this node.

  def <<(c)
    @children << c
    @total_weight = nil
  end

total_weight recursively calculates the weight of this node and everything above it. The @total_weight ||= blah idiom caches the value so we only calculate it once.

  def total_weight
    @total_weight ||= @weight + @children.map {|c| c.total_weight}.sum
  end

balance_weight does the hard work of figuring out the proper weight for the incorrect node by recursively searching through the tree.

  def balance_weight(target=nil)
    by_weight = Hash.new{|h, k| h[k] = []}
    @children.each{|c| by_weight[c.total_weight] << c}

    if by_weight.size == 1 then
      if target
        return @weight - (total_weight - target)
      else
        raise ArgumentError, 'This tree seems balanced!'
      end
    else
      odd_one_out = by_weight.select {|k, v| v.length == 1}.first[1][0]
      child_target = by_weight.select {|k, v| v.length > 1}.first[0]
      return odd_one_out.balance_weight child_target
    end
  end

A couple of utility functions for displaying trees finish off the class.

  def to_s
    "#{@name} (#{@weight})"
  end

  def print_tree(n=0)
    puts "#{'    '*n}#{self} -> #{self.total_weight}"
    @children.each do |child|
      child.print_tree n+1
    end
  end

end

build_circus takes input as a list of lists [name, weight, children]. We make two passes over this list, first creating all the nodes, then building the tree by adding children to parents.

def build_circus(data)
  all_nodes = {}
  all_children = Set.new

  data.each do |name, weight, children|
    all_nodes[name] = CircusNode.new name, weight
  end

  data.each do |name, weight, children|
    children.each {|child| all_nodes[name] << all_nodes[child]}
    all_children.merge children
  end

  root_name = (all_nodes.keys.to_set - all_children).first
  return all_nodes[root_name]
end

Finally, build the tree and solve the problem! Note that we use String.to_sym to convert the node names to symbols (written in Ruby as :symbol), because they’re faster to work with in Hashes and Sets as we do above.

data = readlines.map do |line|
  match = /(?<parent>\w+) \((?<weight>\d+)\)(?: -> (?<children>.*))?/.match line
  [match['parent'].to_sym,
   match['weight'].to_i,
   match['children'] ? match['children'].split(', ').map {|x| x.to_sym} : []]
end

root = build_circus data

puts "Root node: #{root}"

puts root.balance_weight

Memory Reallocation — Python — #adventofcode Day 6

Sat, 09 Dec 2017 17:45:46 +0000

Today’s challenge asks us to follow a recipe for redistributing objects in memory that bears a striking resemblance to the rules of the African game Mancala.

→ Full code on GitHub

!!! commentary When I was doing my MSci, one of our programming exercises was to write (in Haskell, IIRC) a program to play a Mancala variant called Oware, so this had a nice ring of nostalgia.

Back to Python today: it's already become clear that it's by far my most fluent language, which makes sense as it's the only one I've used consistently since my schooldays. I'm a bit behind on the blog posts, so you get this one without any explanation, for now at least!

import math


def reallocate(mem):
    max_val = -math.inf
    size = len(mem)
    for i, x in enumerate(mem):
        if x > max_val:
            max_val = x
            max_index = i

    i = max_index
    mem[i] = 0
    remaining = max_val

    while remaining > 0:
        i = (i + 1) % size
        mem[i] += 1
        remaining -= 1

    return mem


def detect_cycle(mem):
    mem = list(mem)
    steps = 0
    prev_states = {}

    while tuple(mem) not in prev_states:
        prev_states[tuple(mem)] = steps
        steps += 1
        mem = reallocate(mem)

    return (steps, steps - prev_states[tuple(mem)])


initial_state = map(int, input().split())

print("Initial state is ", initial_state)
steps, cycle = detect_cycle(initial_state)
print("Steps to cycle: ", steps)
print("Steps in cycle: ", cycle)

A Maze of Twisty Trampolines — C++ — #adventofcode Day 5

Wed, 06 Dec 2017 17:49:53 +0000

Today’s challenge has us attempting to help the CPU escape from a maze of instructions. It’s not quite a Turing Machine, but it has that feeling of moving a read/write head up and down a tape acting on and changing the data found there.

→ Full code on GitHub

!!! commentary I haven’t written anything in C++ for over a decade. It sounds like there have been lots of interesting developments in the language since then, with C++11, C++14 and the freshly finalised C++17 standards (built-in parallelism in the STL!). I won’t use any of those, but I thought I’d dust off my C++ and see what happened. Thankfully the Standard Template Library classes still did what I expected!

As usual, we first include the parts of the standard library we’re going to use: iostream for input & output; vector for the container. We also declare that we’re using the std namespace, so that we don’t have to prepend vector and the other classes with std::.

#include <iostream>
#include <vector>

using namespace std;

steps_to_escape_part1 implements part 1 of the challenge: we read a location, move forward/backward by the number of steps given in that location, then add one to the location before repeating. The result is the number of steps we take before jumping outside the list.

int steps_to_escape_part1(vector<int>& instructions) {
  int pos = 0, iterations = 0, new_pos;

  while (pos < instructions.size()) {
    new_pos = pos + instructions[pos];
    instructions[pos]++;
    pos = new_pos;
    iterations++;
  }

  return iterations;
}

steps_to_escape_part2 solves part 2, which is very similar, except that an offset greater than 3 is decremented instead of incremented before moving on.

int steps_to_escape_part2(vector<int>& instructions) {
  int pos = 0, iterations = 0, new_pos, offset;

  while (pos < instructions.size()) {
    offset = instructions[pos];
    new_pos = pos + offset;
    instructions[pos] += offset >=3 ? -1 : 1;
    pos = new_pos;
    iterations++;
  }

  return iterations;
}

Finally we pull it all together and link it up to the input.

int main() {
  vector<int> instructions1, instructions2;
  int n;

The cin class lets us read data from standard input, which we then add to a vector of ints to give our list of instructions.

  while (true) {
    cin >> n;
    if (cin.eof())
      break;
    instructions1.push_back(n);
  }

Solving the problem modifies the input, so we need to take a copy to solve part 2 as well. Thankfully the STL makes this easy with iterators.

  instructions2.insert(instructions2.begin(),
                       instructions1.begin(), instructions1.end());

Finally, compute the result and print it on standard output.

  cout << steps_to_escape_part1(instructions1) << endl;
  cout << steps_to_escape_part2(instructions2) << endl;

  return 0;
}

High Entropy Passphrases — Python — #adventofcode Day 4

Tue, 05 Dec 2017 21:18:20 +0000

Today’s challenge describes some simple rules supposedly intended to enforce the use of secure passwords. All we have to do is test a list of passphrase and identify which ones meet the rules.

→ Full code on GitHub

!!! commentary Fearing that today might be as time-consuming as yesterday, I returned to Python and it’s hugely powerful “batteries-included” standard library. Thankfully this challenge was more straightforward, and I actually finished this before finishing day 3.

First, let’s import two useful utilities.

from fileinput import input
from collections import Counter

Part 1 requires simply that a passphrase contains no repeated words. No problem: we split the passphrase into words and count them, and check if any was present more than once.

Counter is an amazingly useful class to have in a language’s standard library. All it does is count things: you add objects to it, and then it will tell you how many of a given object you have. We’re going to use it to count those potentially duplicated words.

def is_valid(passphrase):
    counter = Counter(passphrase.split())
    return counter.most_common(1)[0][1] == 1

Part 2 requires that no word in the passphrase be an anagram of any other word. Since we don’t need to do anything else with the words afterwards, we can check for anagrams by sorting the letters in each word: “leaf” and “flea” both become “aefl” and can be compared directly. Then we count as before.

def is_valid_ana(passphrase):
    counter = Counter(''.join(sorted(word)) for word in passphrase.split())
    return counter.most_common(1)[0][1] == 1

Finally we pull everything together. sum(map(boolean_func, list)) is a common idiom in Python for counting the number of times a condition (checked by boolean_func) is true. In Python, True and False can be treated as the numbers 1 and 0 respectively, so that summing a list of Boolean values gives you the number of True values in the list.

lines = list(input())
print(sum(map(is_valid, lines)))
print(sum(map(is_valid_ana, lines)))

Spiral Memory — Go — #adventofcode Day 3

Tue, 05 Dec 2017 18:06:52 +0000

Today’s challenge requires us to perform some calculations on an “experimental memory layout”, with cells moving outwards from the centre of a square spiral (squiral?).

→ Full code on GitHub

!!! commentary I’ve been wanting to try my hand at Go, the memory-safe, statically typed compiled language from Google for a while. Today’s challenge seemed a bit more mathematical in nature, meaning that I wouldn’t need too many advanced language features or knowledge of a standard library, so I thought I’d give it a “go”. It might have been my imagination, but it was impressive how quickly the compiled program chomped through 60 different input values while I was debugging.

I actually spent far too long on this problem because my brain led me down a blind alley trying to do the wrong calculation, but I got there in the end! The solution is a bit difficult to explain without diagrams, which I don't really have time to draw right now, but fear not because several other people have. First take a look at [the challenge itself which explains the spiral memory concept](http://adventofcode.com/2017/day/3). Then look at the [nice diagrams that Phil Tooley made with Python](http://acceleratedscience.co.uk/blog/adventofcode-day-3-spiral-memory/) and hopefully you'll be able to see what's going on!

It's interesting to note that this challenge also admits of an algorithmic solution instead of the mathematical one: you can model the memory as an infinite grid using a suitable data structure and literally move around it in a spiral. In hindsight this is a much better way of solving the challenge quickly because it's easier and less error-prone to code. I'm quite pleased with my maths-ing though, and it's much quicker than the algorithmic version!

First some Go boilerplate: we have to define the package we’re in (main, because it’s an executable we’re producing) and import the libraries we’ll use.

package main

import (
	"fmt"
	"math"
	"os"
)

Weirdly, Go doesn’t seem to have these basic mathematics functions for integers in its standard library (please someone correct me if I’m wrong!) so I’ll define them instead of mucking about with data types. Go doesn’t do any implicit type conversion, even between numeric types, and the math builtin package only operates on float64 values.

func abs(n int) int {
	if n < 0 {
		return -n
	}
	return n
}

func min(x, y int) int {
	if x < y {
		return x
	}
	return y
}

func max(x, y int) int {
	if x > y {
		return x
	}
	return y
}

This does the heavy lifting for part one: converting from a position on the spiral to a column and row in the grid. (0, 0) is the centre of the spiral. This actually does a bit more than is necessary to calculate the distance as required for part 1, but we’ll use it again for part 2.

func spiral_to_xy(n int) (int, int) {
	if n == 1 {
		return 0, 0
	}

	r := int(math.Floor((math.Sqrt(float64(n-1)) + 1) / 2))
	n_r := n - (2*r-1)*(2*r-1)
	o := ((n_r - 1) % (2 * r)) - r + 1
	sector := (n_r - 1) / (2 * r)

	switch sector {
	case 0:
		return r, o
	case 1:
		return -o, r
	case 2:
		return -r, -o
	case 3:
		return o, -r
	}

	return 0, 0
}

Now use spiral_to_xy to calculate the Manhattan distance that the value at location n in the spiral memory are carried to reach the “access port” at 0.

func distance(n int) int {
	x, y := spiral_to_xy(n)

	return abs(x) + abs(y)
}

This function does the opposite of spiral_to_xy, translating a grid position back to its position on the spiral. This is the one that took me far too long to figure out because I had a brain bug and tried to calculate the value s (which sector or quarter of the spiral we’re looking at) in a way that was never going to work! Fortunately I came to my senses.

func xy_to_spiral(x, y int) int {
	if x == 0 && y == 0 {
		return 1
	}

	r := max(abs(x), abs(y))
	var s, o, n int

	if x+y > 0 && x-y >= 0 {
		s = 0
	} else if x-y < 0 && x+y >= 0 {
		s = 1
	} else if x+y < 0 && x-y <= 0 {
		s = 2
	} else {
		s = 3
	}

	switch s {
	case 0:
		o = y
	case 1:
		o = -x
	case 2:
		o = -y
	case 3:
		o = x
	}

	n = o + r*(2*s+1) + (2*r-1)*(2*r-1)

	return n
}

This is a utility function that uses xy_to_spiral to fetch the value at a given (x, y) location, and returns zero if we haven’t filled that location yet.

func get_spiral(mem []int, x, y int) int {
	n := xy_to_spiral(x, y) - 1
	if n < len(mem) {
		return mem[n]
	}

	return 0
}

Finally we solve part 2 of the problem, which involves going round the spiral writing values into it that are the sum of some values already written. The result is the first of these sums that is greater than or equal to the given input value.

func stress_test(input int) int {
	mem := make([]int, 1)
	n := 0
	mem[0] = 1

	for mem[n] < input {
		n++
		x, y := spiral_to_xy(n + 1)
		mem = append(mem,
			get_spiral(mem, x+1, y)+
				get_spiral(mem, x+1, y+1)+
				get_spiral(mem, x, y+1)+
				get_spiral(mem, x-1, y+1)+
				get_spiral(mem, x-1, y)+
				get_spiral(mem, x-1, y-1)+
				get_spiral(mem, x, y-1)+
				get_spiral(mem, x+1, y-1))
	}

	return mem[n]
}

Now the last part of the program puts it all together, reading the input value from a commandline argument and printing the results of the two parts of the challenge:

func main() {
	var n int
	fmt.Sscanf(os.Args[1], "%d", &n)

	fmt.Printf("Input is %d\n", n)
	fmt.Printf("Distance is %d\n", distance(n))
	fmt.Printf("Stress test result is %d\n", stress_test(n))
}

Corruption Checksum — Python — #adventofcode Day 2

Sat, 02 Dec 2017 09:15:46 +0000

Today’s challenge is to calculate a rather contrived “checksum” over a grid of numbers.

→ Full code on GitHub

!!! commentary Today I went back to plain Python, and I didn’t do formal tests because only one test case was given for each part of the problem. I just got stuck in.

I did write part 2 out in as nested `for` loops as an intermediate step to working out the generator expression. I think that expanded version may have been more readable.

Having got that far, I couldn't then work out how to finally eliminate the need for an auxiliary function entirely without either sorting the same elements multiple times or sorting each row as it's read.

First we read in the input, split it and convert it to numbers. fileinput.input() returns an iterator over the lines in all the files passed as command-line arguments, or over standard input if no files are given.

from fileinput import input
sheet = [[int(x) for x in l.split()] for l in input()]

Part 1 of the challenge calls for finding the difference between the largest and smallest number in each row, and then summing those differences:

print(sum(max(x) - min(x) for x in sheet))

Part 2 is a bit more involved: for each row we have to find the unique pair of elements that divide into each other without remainder, then sum the result of those divisions. We can make it a little easier by sorting each row; then we can take each number in turn and compare it only with the numbers after it (which are guaranteed to be larger). Doing this ensures we only make each comparison once.

def rowsum_div(row):
    row = sorted(row)
    return sum(y // x for i, x in enumerate(row) for y in row[i+1:] if y % x == 0)

print(sum(map(rowsum_div, sheet)))

We can make this code shorter (if not easier to read) by sorting each row as it’s read:

sheet = [sorted(int(x) for x in l.split()) for l in input()]

Then we can just use the first and last elements in each row for part 1, as we know those are the smallest and largest respectively in the sorted row:

print(sum(x[-1] - x[0] for x in sheet))

Part 2 then becomes a sum over a single generator expression:

print(sum(y // x for row in sheet
          for i, x in enumerate(row)
          for y in row[i+1:]
          if y % x == 0))

Very satisfying!

Inverse Captcha — Coconut — #adventofcode Day 1

Fri, 01 Dec 2017 14:35:40 +0000

Well, December’s here at last, and with it Day 1 of Advent of Code.

… It goes on to explain that you may only leave by solving a captcha to prove you’re not a human. Apparently, you only get one millisecond to solve the captcha: too fast for a normal human, but it feels like hours to you. …

As well as posting solutions here when I can, I’ll be putting them all on https://github.com/jezcope/aoc2017 too.

!!! commentary After doing some challenges from last year in Haskell for a warm up, I felt inspired to try out the functional-ish Python dialect, Coconut. Now that I’ve done it, it feels a bit of an odd language, neither fish nor fowl. It’ll look familiar to any Pythonista, but is loaded with features normally associated with functional languages, like pattern matching, destructuring assignment, partial application and function composition.

That makes it quite fun to work with, as it works similarly to Haskell, but because it's restricted by the basic rules of Python syntax everything feels a bit more like hard work than it should.

The accumulator approach feels clunky, but it's necessary to allow [tail call elimination](https://en.wikipedia.org/wiki/Tail_call), which Coconut will do and I wanted to see in action. Lo and behold, if you take a look at the [compiled Python version](https://github.com/jezcope/aoc2017/blob/86c8100824bda1b35e5db6e02d4b80890be7a022/01-inverse-captcha.py#L675) you'll see that my recursive implementation has been turned into a non-recursive `while` loop.

Then again, maybe I'm just jealous of Phil Tooley's [one-liner solution in Python](https://github.com/ptooley/aocGolf/blob/1380d78194f1258748ccfc18880cfd575baf5d37/2017.py#L8).

import sys

def inverse_captcha_(s, acc=0):
    case reiterable(s):
        match (|d, d|) :: rest:
            return inverse_captcha_((|d|) :: rest, acc + int(d))
        match (|d0, d1|) :: rest:
            return inverse_captcha_((|d1|) :: rest, acc)

    return acc


def inverse_captcha(s) = inverse_captcha_(s :: s[0])


def inverse_captcha_1_(s0, s1, acc=0):
    case (reiterable(s0), reiterable(s1)):
        match ((|d0|) :: rest0, (|d0|) :: rest1):
            return inverse_captcha_1_(rest0, rest1, acc + int(d0))
        match ((|d0|) :: rest0, (|d1|) :: rest1):
            return inverse_captcha_1_(rest0, rest1, acc)

    return acc


def inverse_captcha_1(s) = inverse_captcha_1_(s, s$[len(s)//2:] :: s)


def test_inverse_captcha():
    assert "1111" |> inverse_captcha == 4
    assert "1122" |> inverse_captcha == 3
    assert "1234" |> inverse_captcha == 0
    assert "91212129" |> inverse_captcha == 9


def test_inverse_captcha_1():
    assert "1212" |> inverse_captcha_1 == 6
    assert "1221" |> inverse_captcha_1 == 0
    assert "123425" |> inverse_captcha_1 == 4
    assert "123123" |> inverse_captcha_1 == 12
    assert "12131415" |> inverse_captcha_1 == 4

if __name__ == "__main__":
    sys.argv[1] |> inverse_captcha |> print
    sys.argv[1] |> inverse_captcha_1 |> print

Advent of Code 2017: introduction

Wed, 29 Nov 2017 18:33:08 +0000

It’s a common lament of mine that I don’t get to write a lot of code in my day-to-day job. I like the feeling of making something from nothing, and I often look for excuses to write bits of code, both at work and outside it.

Advent of Code is a daily series of programming challenges for the month of December, and is about to start its third annual incarnation. I discovered it too late to take part in any serious way last year, but I’m going to give it a try this year.

There are no restrictions on programming language (so of course some people delight in using esoteric languages like Brainf**k), but I think I’ll probably stick with Python for the most part. That said, I miss my Haskell days and I’m intrigued by new kids on the block Go and Rust, so I might end up throwing in a few of those on some of the simpler challenges.

I’d like to focus a bit more on how I solve the puzzles. They generally come in two parts, with the second part only being revealed after successful completion of the first part. With that in mind, test-driven development makes a lot of sense, because I can verify that I haven’t broken the solution to the first part in modifying to solve the second.

I may also take a literate programming approach with org-mode or Jupyter notebooks to document my solutions a bit more, and of course that will make it easier to publish solutions here so I’ll do that as much as I can make time for.

On that note, here are some solutions for 2016 that I’ve done recently as a warmup.

Day 1: Python

Day 1 instructions

import numpy as np
import pytest as t
import sys

TURN = {
    'L': np.array([[0,  1],
                   [-1, 0]]),
    'R': np.array([[0, -1],
                   [1,  0]])
}
ORIGIN = np.array([0, 0])
NORTH = np.array([0, 1])


class Santa:
    def __init__(self, location, heading):
        self.location = np.array(location)
        self.heading = np.array(heading)
        self.visited = [(0,0)]

    def execute_one(self, instruction):
        start_loc = self.location.copy()
        self.heading = self.heading @ TURN[instruction[0]]
        self.location += self.heading * int(instruction[1:])
        self.mark(start_loc, self.location)

    def execute_many(self, instructions):
        for i in instructions.split(','):
            self.execute_one(i.strip())

    def distance_from_start(self):
        return sum(abs(self.location))

    def mark(self, start, end):
        for x in range(min(start[0], end[0]), max(start[0], end[0])+1):
            for y in range(min(start[1], end[1]), max(start[1], end[1])+1):
                if any((x, y) != start):
                    self.visited.append((x, y))

    def find_first_crossing(self):
        for i in range(1, len(self.visited)):
            for j in range(i):
                if self.visited[i] == self.visited[j]:
                    return self.visited[i]

    def distance_to_first_crossing(self):
        crossing = self.find_first_crossing()
        if crossing is not None:
            return abs(crossing[0]) + abs(crossing[1])

    def __str__(self):
        return f'Santa @ {self.location}, heading {self.heading}'


def test_execute_one():
    s = Santa(ORIGIN, NORTH)

    s.execute_one('L1')
    assert all(s.location == np.array([-1, 0]))
    assert all(s.heading == np.array([-1, 0]))

    s.execute_one('L3')
    assert all(s.location == np.array([-1, -3]))
    assert all(s.heading == np.array([0, -1]))

    s.execute_one('R3')
    assert all(s.location == np.array([-4, -3]))
    assert all(s.heading == np.array([-1, 0]))

    s.execute_one('R100')
    assert all(s.location == np.array([-4, 97]))
    assert all(s.heading == np.array([0, 1]))


def test_execute_many():
    s = Santa(ORIGIN, NORTH)

    s.execute_many('L1, L3, R3')
    assert all(s.location == np.array([-4, -3]))
    assert all(s.heading == np.array([-1, 0]))

def test_distance():
    assert Santa(ORIGIN, NORTH).distance_from_start() == 0
    assert Santa((10, 10), NORTH).distance_from_start() == 20
    assert Santa((-17, 10), NORTH).distance_from_start() == 27


def test_turn_left():
    east = NORTH @ TURN['L']
    south = east @ TURN['L']
    west = south @ TURN['L']

    assert all(east == np.array([-1, 0]))
    assert all(south == np.array([0, -1]))
    assert all(west == np.array([1, 0]))


def test_turn_right():
    west = NORTH @ TURN['R']
    south = west @ TURN['R']
    east = south @ TURN['R']

    assert all(east == np.array([-1, 0]))
    assert all(south == np.array([0, -1]))
    assert all(west == np.array([1, 0]))


if __name__ == '__main__':
    instructions = sys.stdin.read()

    santa = Santa(ORIGIN, NORTH)
    santa.execute_many(instructions)

    print(santa)
    print('Distance from start:', santa.distance_from_start())
    print('Distance to target: ', santa.distance_to_first_crossing())

Day 2: Haskell

Day 2 instructions

module Main where

data Pos = Pos Int Int deriving (Show)

-- Magrittr-style pipe operator
(|>) :: a -> (a -> b) -> b
x |> f = f x

swapPos :: Pos -> Pos
swapPos (Pos x y) = Pos y x

clamp :: Int -> Int -> Int -> Int
clamp lower upper x | x < lower = lower
                    | x > upper = upper
                    | otherwise = x

clampH :: Pos -> Pos
clampH (Pos x y) = Pos x' y'
  where y' = clamp 0 4 y
        r = abs (2 - y')
        x' = clamp r (4-r) x

clampV :: Pos -> Pos
clampV = swapPos . clampH . swapPos
                        
buttonForPos :: Pos -> String
buttonForPos (Pos x y) = [buttons !! y !! x]
  where buttons = ["  D  ",
                   " ABC ",
                   "56789",
                   " 234 ",
                   "  1  "]

decodeChar :: Pos -> Char -> Pos
decodeChar (Pos x y) 'R' = clampH $ Pos (x+1) y
decodeChar (Pos x y) 'L' = clampH $ Pos (x-1) y
decodeChar (Pos x y) 'U' = clampV $ Pos x (y+1)
decodeChar (Pos x y) 'D' = clampV $ Pos x (y-1)

decodeLine :: Pos -> String -> Pos
decodeLine p "" = p
decodeLine p (c:cs) = decodeLine (decodeChar p c) cs

makeCode :: String -> String
makeCode instructions = lines instructions          -- split into lines
                        |> scanl decodeLine (Pos 1 1) -- decode to positions
                        |> tail                       -- drop start position
                        |> concatMap buttonForPos     -- convert to buttons

main = do
  input <- getContents
  putStrLn $ makeCode input

Research Data Management Forum 18, Manchester

Mon, 20 Nov 2017 09:28:11 +0000

!!! intro "" Monday 20 and Tuesday 21 November 2017 I’m at the Research Data Management Forum in Manchester. I thought I’d use this as an opportunity to try liveblogging, so during the event some notes should appear in the box below (you may have to manually refresh your browser tab periodically to get the latest version).

I've not done this before, so if the blog stops updating then it's probably because I've stopped updating it to focus on the conference instead!

This was made possible using GitHub's cool [Gist](https://gist.github.com) tool.

Draft content policy

Sat, 18 Nov 2017 16:52:20 +0000

I thought it was about time I had some sort of content policy on here so this is a first draft. It will eventually wind up as a separate page. Feedback welcome!

!!! aside “Content policy” This blog’s primary purpose is as a reflective learning tool for my own development; my aim in writing any given post is mainly to expose and develop my own thinking on a topic. My reasons for making a public blog rather than a private journal are:

1. If I'm lucky, someone smarter than me will provide feedback that will help me and my readers to learn more
2. If I'm extra lucky, someone else might learn from the material as well

Each post, therefore, represents the state of my thinking at the time I wrote it, or perhaps a deliberate provocation or exaggeration; either way, if you don't know me personally please don't judge me based entirely on my past words. This is a request though, not an attempt to excuse bad behaviour on my part. I accept full responsibility for any consequences of my words, whether intended or not.

I will not remove comments or ban individuals for disagreeing with me, only for behaving offensively or disrespectfully. I will do my best to be fair and balanced and explain decisions that I take, but I reserve the right to take those decisions without making any explanation at all if it seems likely to further inflame a situation. If I end up responding to anything simply with a link to this policy, that's probably all the explanation you're going to get.

It should go without saying, but the opinions presented in this blog are my own and not those of my employer or anyone else I might at times represent.

Learning to live with anxiety

Sun, 05 Nov 2017 18:03:01 +0000

!!! intro "" This is a post that I’ve been writing for months, and writing in my head for years. For some it will explain aspects of my personality that you might have wondered about. For some it will just be another person banging on self-indulgently about so-called “mental health issues”. Hopefully, for some it will demystify some stuff and show that you’re not alone and things do get better.

For as long as I can remember I’ve been a worrier. I’ve also suffered from bouts of what I now recognise as depression, on and off since my school days. It’s only relatively recently that I’ve come to the realisation that these two might be connected and that my ‘worrying’ might in fact be outside the normal range of healthy human behaviour and might more accurately be described as chronic anxiety. You probably won’t have noticed it, but it’s been there.

More recently I’ve begun feeling like I’m getting on top of it and feeling “normal” for the first time in my life. Things I’ve found that help include: getting out of the house more and socialising with friends; and getting a range of exercise, outdoors and away from the city (rock climbing is mentally and physically engaging and open water swimming is indescribably joyful).

But mostly it’s the cognitive behavioural therapy (CBT) and the antidepressants.

Before I go any further, a word about drugs (“don’t do drugs, kids”): I’m on the lowest available dose of a common antidepressant. This isn’t because it stops me being sad all the time (I’m not) or because it makes all my problems go away (it really doesn’t). It’s because the scientific evidence points to a combination of CBT and antidepressants as being the single most effective treatment for generalised anxiety disorder. The reason for this is simple: CBT isn’t easy, because it asks you to challenge habits and beliefs you’ve held your whole life. In the short term there is going to be more anxiety and some antidepressants are also effective at blunting the effect of this additional anxiety. In short, CBT is what makes you better, and the drugs just make it a little bit more effective.

A lot of people have misconceptions about what it means to be ‘in therapy’. I suspect a lot of these are derived from the psychoanalysis we often see portrayed in (primarily US) film and TV. The problem with that type of navel-gazing therapy is that you can spend years doing it, finally reach some sort of breakthrough insight, and still not have no idea what the supposed insight means for your actual life.

CBT is different in that rather than addressing feelings directly it focuses on habits in your thoughts (cognitive) and actions (behavioural) with feeling better as an outcome (therapy). CBT and related forms of therapy now have decades of clinical evidence showing that they really work.

It uses a wide range of techniques to identify, challenge and reduce various common unhelpful thoughts and behaviours. By choosing and practicing these, you can break bad mental habits that you’ve been carrying around, often for decades.

For me this means giving fair weight to my successes as well as my failings, allowing flexibility into the rigid rules that I have always, subconsciously, lived by, and being a bit kinder to myself when I make mistakes. It’s not been easy and I have to remind myself to practice this every day, but it’s really helped.

!!! aside “More info” If you live in the UK, you might not be aware that you can get CBT and other psychological therapies on the NHS through a scheme called IAPT (improving access to psychological therapies). You can self-refer so you don’t need to see a doctor first, but you might want to anyway if you think medication might help. They also have a progression of treatments, so you might be offered a course of “guided self-help” and then progressed to CBT or another talking therapy if need be. This is what happened to me, and it did help a bit but it was CBT that helped me the most.

Becoming a librarian

Sat, 04 Nov 2017 19:38:01 +0000

What is a librarian? Is it someone who has a masters degree in librarianship and information science? Is it someone who looks after information for other people? Is it simply someone who works in a library?

I’ve been grappling with this question a lot lately because I’ve worked in academic libraries for about 3 years now and I never really thought that’s something that might happen. People keep referring to me as “a librarian” but there’s some imposter feelings here because all the librarians around me have much more experience, have skills in areas like cataloguing and collection management and, generally, have a librarian masters degree.

So I’ve been thinking about what it actually means to me to be a librarian or not.

NB. some of these may be tongue-in-cheek

Ways in which I am a librarian:

I work in a library
I help people to access and organise information
I have a cat
I like gin

Ways in which I am not a librarian:

I don’t have a librarianship qualification
I don’t work with books 😉
I don’t knit (though I can probably remember how if pressed)
I don’t shush people or wear my hair in a bun (I can confirm that this is also true of every librarian I know)

Ways in which I am a shambrarian:

I like beer
I have more IT experience and qualification than librarianship

At the end of the day, I still don’t know how I feel about this or, for that matter, how important it is. I’m probably going to accept whatever title people around me choose to bestow, though any label will chafe at times!

Better Science Through Better Data #scidata17

Mon, 30 Oct 2017 07:21:16 +0000

Better Science through Better DoughnutsJez Cope

Update: fixed the link to the slides so it works now!

Last week I had the honour of giving my first ever keynote talk, at an event entitled Better Science Through Better Data hosted jointly by Springer Nature and the Wellcome Trust. It was nerve-wracking but exciting and seemed to go down fairly well. I even got accidentally awarded a PhD in the programme — if only it was that easy! The slides for the talk, “Supporting Open Research: The role of an academic library”, are available online (doi:10.15131/shef.data.5537269), and the whole event was video’d for posterity and viewable online.

I got some good questions too, mainly from the clever online question system. I didn’t get to answer all of them, so I’m thinking of doing a blog post or two to address a few more.

There were loads of other great presentations as well, both keynotes and 7-minute lightning talks, so I’d encourage you to take a look at at least some of it. I’ll pick out a few of my highlights.

Dr Aled Edwards (University of Toronto)

There’s a major problem with science funding that I hadn’t really thought about before. The available funding pool for research is divided up into pots by country, and often by funding body within a country. Each of these pots have robust processes to award funding to the most important problems and most capable researchers. The problem comes because there is no coordination between these pots, so researchers all over the world end up getting funded to research the most popular problems leading to a lot of duplication of effort.

Industry funding suffers from a similar problem, particularly the pharmaceutical industry. Because there is no sharing of data or negative results, multiple companies spend billions researching the same dead ends chasing after the same drugs. This is where the astronomical costs of drug development come from.

Dr Edwards presented one alternative, modelled by a company called M4K Pharma. The idea is to use existing IP laws to try and give academic researchers a reasonable, morally-justifiable and sustainable profit on drugs they develop, in contrast to the current model where basic research is funded by governments while large corporations hoover up as much profit as they possibly can. This new model would develop drugs all the way to human trial within academia, then license the resulting drugs to companies to manufacture with a price cap to keep the medicines affordable to all who need them.

Core to this effort is openness with data, materials and methodology, and Dr Edwards presented several examples of how this approach benefited academic researchers, industry and patients compared with a closed, competitive focus.

Dr Kirstie Whitaker (Alan Turing Institute)

This was a brilliant presentation, presenting a practical how-to guide to doing reproducible research, from one researcher to another. I suggest you take a look at her slides yourself: Showing your working: a how-to guide to reproducible research.

Dr Whitaker briefly addressed a number of common barriers to reproducible research:

Is not considered for promotion: so it should be!
Held to higher standards than others: reviewers should be discouraged from nitpicking just because the data/code/whatever is available (true unbiased peer review of these would be great though)
Publication bias towards novel findings: it is morally wrong to not publish reproductions, replications etc. so we need to address the common taboo on doing so
Plead the 5th: if you share, people may find flaws, but if you don’t they can’t — if you’re worried about this you should ask yourself why!
Support additional users: some (much?) of the burden should reasonably on the reuser, not the sharer
Takes time: this is only true if you hack it together after the fact; if you do it from the start, the whole process will be quicker!
Requires additional skills: important to provide training, but also to judge PhD students on their ability to do this, not just on their thesis & papers

The rest of the presentation, the “how-to” guide of the title’ was a well-chosen and passionately delivered set of recommendations, but the thing that really stuck out for me is how good Dr Whitaker is at making the point that you only have to do one of these things to improve the quality of your research. It’s easy to get the impression at the moment that you have to be fully, perfectly open or not at all, but it’s actually OK to get there one step at a time, or even not to go all the way at all!

Anyway, I think this is a slide deck that speaks for itself, so I won’t say any more!

Lightning talk highlights

There was plenty of good stuff in the lightning talks, which were constrained to 7 minutes each, but a few of the things that stood out for me were, in no particular order:

Code Ocean — share and run code in the cloud
dat project — peer to peer data syncronisation tool
- Can automate metadata creation, data syncing, versioning
- Set up a secure data sharing network that keeps the data in sync but off the cloud
Berlin Institute of Health — open science course for students
- Pre-print paper
- Course materials
InterMine — taking the pain out of data cleaning & analysis
Nix/NixOS as a component of a reproducible paper
BoneJ (ImageJ plugin for bone analysis) — developed by a scientist, used a lot, now has a Wellcome-funded RSE to develop next version
ESASky — amazing live, online archive of masses of astronomical data

Coda

I really enjoyed the event (and the food was excellent too). My thanks go out to:

The programme committee for asking me to come and give my take — I hope I did it justice!
The organising team who did a brilliant job of keeping everything running smoothly before and during the event
The University of Sheffield for letting me get away with doing things like this!

Blog platform switch

Sun, 29 Oct 2017 21:40:00 +0000

I’ve just switched my blog over to the Nikola static site generator. Hopefully you won’t notice a thing, but there might be a few weird spectres around til I get all the kinks ironed out.

I’ve made the switch for a couple of main reasons:

Nikola supports Jupyter notebooks as a source format for blog posts, which will be useful to include code snippets
It’s written in Python, a language which I actually know, so I’m more likely to be able to fix things that break, customise it and potentially contribute to the open source project (by contrast, Hugo is written in Go, which I’m not really familiar with)

Lean Libraries: applying agile practices to library services

Wed, 19 Jul 2017 17:42:16 +0100

Kanban board Jeff Lasovski (via Wikimedia Commons)

I’ve been working with our IT services at work quite closely for the last year as product owner for our new research data portal, ORDA. That’s been a fascinating process for me as I’ve been able to see first-hand some of the agile techniques that I’ve been reading about from time-to-time on the web over the last few years.

They’re in the process of adopting a specific set of practices going under the name “Scrum”, which is fun because it uses some novel terminology that sounds pretty weird to non-IT folks, like “scrum master”, “sprint” and “product backlog”. On my small project we’ve had great success with the short cycle times and been able to build trust with our stakeholders by showing concrete progress on a regular basis.

Modern librarianship is increasingly fluid, particularly in research services, and I think that to handle that fluidity it’s absolutely vital that we are able to work in a more agile way. I’m excited about the possibilities of some of these ideas. However, Scrum as implemented by our IT services doesn’t seem something that transfers directly to the work that we do: it’s too specialised for software development to adapt directly.

What I intend to try is to steal some of the individual practices on an experimental basis and simply see what works and what doesn’t. The Lean concepts currently popular in IT were originally developed in manufacturing: if they can be translated from the production of physical goods to IT, I don’t see why we can’t make the ostensibly smaller step of translating them to a different type of knowledge work.

I’ve therefore started reading around this subject to try and get as many ideas as possible. I’m generally pretty rubbish at taking notes from books, so I’m going to try and record and reflect on any insights I make on this blog. The framework for trying some of these out is clearly a Plan-Do-Check-Act continuous improvement cycle, so I’ll aim to reflect on that process too.

I’m sure there will have been people implementing Lean in libraries already, so I’m hoping to be able to discover and learn from them instead of starting froms scratch. Wish me luck!

Mozilla Global Sprint 2017

Thu, 15 Jun 2017 07:52:23 +0100

Photo by Lena Bell on Unsplash

Every year, the Mozilla Foundation runs a two-day Global Sprint, giving people around the world 50 hours to work on projects supporting and promoting open culture and tech.

Though much of the work during the sprint is, of course, technical software development work, there are always tasks suited to a wide range of different skill sets and experience levels. The participants include writers, designers, teachers, information professionals and many others.

This year, for the first time, the University of Sheffield hosted a site, providing a space for local researchers, developers and others to get out of their offices, work on #mozsprint and link up with others around the world. The Sheffield site was organised by the Research Software Engineering group in collaboration with the University Library. Our site was only small compared to others, but we still had people working on several different projects.

My reason for taking part in the sprint was to contribute to the international effort on the Library Carpentry project. A team spread across four continents worked throughout the whole sprint to review and develop our lesson material.

As there were no other Library Carpentry volunteers at the Sheffield site, I chose to work on some urgent work around improving the presentation of our workshops and lessons on the web and related workflows. It was a really nice subproject to work on, requiring not only cleaning up and normalising the metadata we hold on workshops and lessons, but also digesting and formalising our current ad hoc process of lesson development.

The largest group were solar physicists from the School of Maths and Statistics, working on the SunPy project, an open source environment for solar data analysis. They pushed loads of bug fixes and documentation improvements, and also mentored a new contributor through their first additions to the project.

Anna Krystalli from Research Software Engineering worked on the EchoBurst project, which is building a web browser extension to help people break out of their online echo chambers. It does this by using natural language processing techniques to highlight well-written, logically sound articles that disagree with the reader’s stated views on particular topics of interest. Anna was part of an effort to begin extending this technology to online videos.

We had a couple of individuals simply taking the opportunity to break out of their normal work environments to work or learn, including a couple of members of library staff show up for a couple of hours to learn how to use git on a new project!

Not just certifiable…

Fri, 05 May 2017 17:19:18 +0100

A couple of months ago, I went to Oxford for an intensive, 2-day course run by Software Carpentry and Data Carpentry for prospective new instructors.

I’ve now had confirmation that I’ve completed the checkout procedure so it’s official: I’m now a certified Data Carpentry instructor! As far as I’m aware, the certification process is now combined, so I’m also approved to teach Software Carpentry material too. And of course there’s Library Carpentry too…

IDCC 2017 reflection

Thu, 06 Apr 2017 07:26:22 +0100

For most of the last few years I've been lucky enough to attend the International Digital Curation Conference (IDCC). One of the main audiences attending is people who, like me, work on research data management at universities around the world and it's begun to feel like a sort of "home" conference to me. This year, IDCC was held at the Royal College of Surgeons in the beautiful city of Edinburgh.

For the last couple of years, my overall impression has been that, as a community, we're moving away from the "first-order" problem of trying to convince people (from PhD students to senior academics) to take RDM seriously and into a rich set of "second-order" problems around how to do things better and widen support to more people. This year has been no exception. Here are a few of my observations and takeaway points.

Everyone has a repository now: Only last year, the most common question you'd get asked by strangers in the coffee break would be "Do you have a data repository?" Now the question is more likely to be "What are you using for your data repository?", along with more subtle questions about specific components of systems and how they interact.
Integrating active storage and archival systems: Now that more institutions have data worth preserving, there is more interest in (and in many cases experience of) setting up more seamless integrations between active and archival storage. There are lessons here we can learn.
Freezing in amber vs actively maintaining assets: There seemed to be an interesting debate going on throughout the conference around the aim of preservation: should we be faithfully preserving the bits and bytes provided without trying to interpret them, or should we take a more active approach by, for example, migrating obsolete formats to newer alternatives. If the former, should we attempt to preserve the software required to access the data as well? If the latter, how much effort do we invest and how do we ensure nothing is lost or altered in the migration?
Demonstrating Data Science instead of debating what it is: The phrase "Data Science" was once again one of the most commonly uttered of the conference. However, there is now less abstract discussion about what, exactly, is meant by this "data science" thing; this has been replaced more by concrete demonstrations. This change was exemplified perfectly by the keynote by data scientist Alice Daish, who spent a riveting 40 minutes or so enthusing about all the cool stuff she does with data at the British Museum.
Recognition of software as an issue: Even as recently as last year, I've struggled to drum up much interest in discussing software sustainability and preservation at events like this; the interest was there, but there were higher priorities. So I was completely taken by surprise when we ended up with 30+ people in the Software Preservation Birds of a Feather (BoF) session, and when very little input was needed from me as chair to keep a productive discussion going for a full 90 minutes.
Unashamed promotion of openness: As a community we seem to have nearly overthrown our collective embarrassment about the phrase "open data" (although maybe this is just me). We've always known it was a good thing, but I know I've been a bit of an apologist in the past, feeling that I had to "soften the blow" when asking researchers to be more open. Now I feel more confident in leading with the benefits of openness, and it felt like that's a change reflected in the community more widely.
Becoming more involved in the conference: This year, I took a decision to try and do more to contribute to the conference itself, and I felt like this was pretty successful both in making that contribution and building up my own profile a bit. I presented a paper on one of my current passions, Library Carpentry; it felt really good to be able to share my enthusiasm. I presented a poster on our work integrating our data repository and digital preservation platform; this gave me more of a structure for networking during breaks, as I was able to stand by the poster and start discussions with anyone who seemed interested. I chaired a parallel session; a first for me, and a different challenge from presenting or simply attending the talks. And finally, I proposed and chaired the Software Preservation BoF session (blog post forthcoming).
Renewed excitement: It's weird, and possibly all in my imagination, but there seemed to be more energy at this conference than at the previous couple I've been to. More people seemed to be excited about the work we're all doing, recent achievements and the possibilities for the future.

Chat rooms vs Twitter: how I communicate now

Wed, 29 Mar 2017 21:41:31 +0100

CC0, Pixabay

This time last year, Brad Colbow published a comic in his “The Brads” series entitled “The long slow death of Twitter”. It really encapsulates the way I’ve been feeling about Twitter for a while now.

Go ahead and take a look. I’ll still be here when you come back.

According to my Twitter profile, I joined in February 2009 as user #20,049,102. It was nearing its 3rd birthday and, though there were clearly a lot of people already signed up at that point, it was still relatively quiet, especially in the UK.

I was a lonely PhD student just starting to get interested in educational technology, and one thing that Twitter had in great supply was (and still is) people pushing back the boundaries of what tech can do in different contexts.

Somewhere along the way Twitter got really noisy, partly because more people (especially commercial companies) are using it more to talk about stuff that doesn’t interest me, and partly because I now follow 1,200+ people and find I get several tweets a second at peak times, which no-one could be expected to handle. More recently I’ve found my attention drawn to more focussed communities instead of that big old shouting match.

I find I’m much more comfortable discussing things and asking questions in small focussed communities because I know who might be interested in what. If I come across an article about a cool new Python library, I’ll geek out about it with my research software engineer friends; if I want advice on an aspect of my emacs setup, I’ll ask a bunch of emacs users.

I feel like I’m talking to people who want to hear what I’m saying. Next to that experience, Twitter just feels like standing on a street corner shouting.

IRC channels (mostly on Freenode), and similar things like Slack and gitter form the bulk of this for me, along with a growing number of WhatsApp group chats. Although online chat is theoretically a synchronous medium, I find that I can treat it more as “semi-synchronous”: I can have real-time conversations as they arise, but I can also close them and tune back in later to catch up if I want. Now I come to think about it, this is how I used to treat Twitter before the 1,200 follows happened.

I also find I visit a handful of forums regularly, mostly of the Reddit link-sharing or StackExchange Q&A type. /r/buildapc was invaluable when I was building my latest box, /r/EarthPorn (very much not NSFW) is just beautiful.

I suppose the risk of all this is that I end up reinforcing my own echo chamber. I’m not sure how to deal with that, but I certainly can’t deal with it while also suffering from information overload.

Introducing PyRefine: OpenRefine meets Python

Mon, 27 Mar 2017 21:00:56 +0100

I’m knocking the rust off my programming skills by attempting to write a pure-Python interpreter for OpenRefine “scripts”.

OpenRefine is a great tool for exploring and cleaning datasets prior to analysing them. It also records an undo history of all actions that you can export as a sort of script in JSON format. One thing that bugs me though is that, having spent some time interactively cleaning up your dataset, you then need to fire up OpenRefine again and do some interactive mouse-clicky stuff to apply that cleaning routine to another dataset. You can at least re-import the JSON undo history to make that as quick as possible, but there’s no getting around the fact that there’s no quick way to do it from a cold start.

There is a project, BatchRefine, that extends the OpenRefine server to accept batch requests over a HTTP API, but that isn’t useful when you can’t or don’t want to keep a full Java stack running in the background the whole time.

My concept is this: you use OR to explore the data interactively and design a cleaning process, but then export the process to JSON and integrate it into your analysis in Python. That way it can be repeated ad nauseam without having to fire up a full Java stack.

I’m taking some inspiration from the great talk “So you want to be a wizard?” by Julia Evans (@b0rk), who recommends trying experiments as a way to learn. She gives these Rules of Programming Experiments:

“it doesn’t have to be good

it doesn’t have to work

you have to learn something”

In that spirit, my main priorities are: to see if this can be done; to see how far I can get implementing it; and to learn something. If it also turns out to be a useful thing, well, that’s a bonus. Some of the interesting possible challenges here:

Implement all core operations; there are quite a lot of these, some of which will be fun (i.e. non-trivial) to implement
Implement (a subset of?) GREL, the General Refine Expression Language; I guess my undergrad course on implementing parsers and compilers will come in handy after all!
Generate clean, sane Python code from the JSON rather than merely executing it; more than anything, this would be a nice educational tool for users of OpenRefine who want to see how to do equivalent things in Python
Selectively optimise key parts of the process; this will involve profiling the code to identify bottlenecks as well as tweaking the actual code to go faster
Potentially handle contributions to the code from other people; I’d be really happy if this happened but I’m realistic…

If you’re interested, the project is called PyRefine and it’s on github. Constructive criticism, issues & pull requests all welcome!

Implementing Yesterbox in emacs with mu4e

Thu, 27 Oct 2016 08:30:33 +0100

I’ve been meaning to give Yesterbox a try for a while. The general idea is that each day you only deal with email that arrived yesterday or earlier. This forms your inbox for the day, hence “yesterbox”.

Once you’ve emptied your yesterbox, or at least got through some minimum number (10 is recommended) then you can look at emails from today. Even then you only really want to be dealing with things that are absolutely urgent. Anything else can wait til tomorrow.

The motivation for doing this is to get away from the feeling that we are King Canute, trying to hold back the tide. I find that when I’m processing my inbox toward zero there’s always a temptation to keep skipping to the new stuff that’s just come in. Hiding away the new email until I’ve dealt with the old is a very interesting idea.

I use mu4e in emacs for reading my email, and handily the mu search syntax is very flexible so you’d think it would be easy to create a yesterbox filter:

maildir:"/INBOX" date:..1d

Unfortunately, 1d is interpreted as “24 hours ago from right now” so this filter misses everything that was sent yesterday but less than 24 hours ago. There was a feature request raised on the mu github repository to implement an additional date filter syntax but it seems to have died a death for now. In the meantime, the answer to this is to remember that my workplace observes fairly standard office hours, so that anything sent more than 9 hours ago is unlikely to have been sent today. The following does the trick:

maildir:"/INBOX" date:..9h

In my mu4e bookmarks list, that looks like this:

(setq mu4e-bookmarks
      '(("flag:unread AND NOT flag:trashed" "Unread messages" ?u)
        ("flag:flagged maildir:/archive" "Starred messages" ?s)
        ("date:today..now" "Today's messages" ?t)
        ("date:7d..now" "Last 7 days" ?w)
        ("maildir:\"/Mailing lists.*\" (flag:unread OR flag:flagged)" "Unread in mailing lists" ?M)
        ("maildir:\"/INBOX\" date:..1d" "Yesterbox" ?y))) ;; <- this is the new one

Rewarding good practice in research

Thu, 13 Oct 2016 08:32:50 +0100

From opensource.com on Flickr

Whenever I’m involved in a discussion about how to encourage researchers to adopt new practices, eventually someone will come out with some variant of the following phrase:

“That’s all very well, but researchers will never do XYZ until it’s made a criterion in hiring and promotion decisions.”

With all the discussion of carrots and sticks I can see where this attitude comes from, and strongly empathise with it, but it raises two main problems:

It’s unfair and more than a little insulting to anyone to be lumped into one homogeneous group; and
Taking all the different possible XYZs into account, that’s an awful lot of hoops to expect anyone to jump through.

Firstly, “researchers” are as diverse as the rest of us in terms of what gets them out of bed in the morning. Some of us want prestige; some want to contribute to a greater good; some want to create new things; some just enjoy the work.

One thing I’d argue we all have in common is this: nothing is more offputting than feeling like you’re being strongarmed into something you don’t want to do.

If we rely on simplistic metrics, people will focus on those and miss the point. At best people will disengage and at worst they will actively game the system. I’ve got to do these ten things to get my next payrise, and still retain my sanity? Ok, what’s the least I can get away with and still tick them off. You see it with students taking poorly-designed assessments and grown-ups are no difference.

We do need to wield carrots as well as sticks, but the whole point is that these practices are beneficial in and of themselves. The carrots are already there if we articulate them properly and clear the roadblocks (don’t you enjoy mixed metaphors?). Creating artificial benefits will just dilute the value of the real ones.

Secondly, I’ve heard a similar argument made for all of the following practices and more:

Research data management
Open Access publishing
Public engagement
New media (e.g. blogging)
Software management and sharing

Some researchers devote every waking hour to their work, whether it’s in the lab, writing grant applications, attending conferences, authoring papers, teaching, and so on and so on. It’s hard to see how someone with all this in their schedule can find time to exercise any of these new skills, let alone learn them in the first place. And what about the people who sensibly restrict the hours taken by work to spend more time doing things they enjoy?

Yes, all of the above practices are valuable, both for the individual and the community, but they’re all new (to most) and hence require more effort up front to learn. We have to accept that it’s inevitably going to take time for all of them to become “business as usual”.

I think if the hiring/promotion/tenure process has any role in this, it’s in asking whether the researcher can build a coherent narrative as to why they’ve chosen to focus their efforts in this area or that. You’re not on Twitter but your data is being used by 200 research groups across the world? Great! You didn’t have time to tidy up your source code for github but your work is directly impacting government policy? Brilliant!

We still need convince more people to do more of these beneficial things, so how? Call me naïve, but maybe we should stick to making rational arguments, calming fears and providing low-risk opportunities to learn new skills. Acting (compassionately) like a stuck record can help. And maybe we’ll need to scale back our expectations in other areas (journal impact factors, anyone?) to make space for the new stuff.

Software Carpentry: SC Test; does your software do what you meant?

Thu, 06 Oct 2016 18:51:42 +0100

“The single most important rule of testing is to do it.”
— Brian Kernighan and Rob Pike, The Practice of Programming (quote taken from SC Test page

One of the trickiest aspects of developing software is making sure that it actually does what it’s supposed to. Sometimes failures are obvious: you get completely unreasonable output or even (shock!) a comprehensible error message.

But failures are often more subtle. Would you notice if your result was out by a few percent, or consistently ignored the first row of your input data?

The solution to this is testing: take some simple example input with a known output, run the code and compare the actual output with the expected one. Implement a new feature, test and repeat. Sounds easy, doesn’t it?

But then you implement a new bit of code. You test it and everything seems to work fine, except that your new feature required changes to existing code and those changes broke something else. So in fact you need to test everything, and do it every time you make a change. Further than that, you probably want to test that all your separate bits of code work together properly (integration testing) as well as testing the individual bits separately (unit testing). In fact, splitting your tests up like that is a good way of holding on to your sanity.

This is actually a lot less scary than it sounds, because there are plenty of tools now to automate that testing: you just type a simple test command and everything is verified. There are even tools that enable you to have tests run automatically when you check the code into version control, and even automatically deploy code that passes the tests, a process known as continuous integration or CI.

The big problems with testing are that it’s tedious, your code seems to work without it and no-one tells you off for not doing it.

At the time when the Software Carpentry competition was being run, the idea of testing wasn’t new, but the tools to help were in their infancy.

“Existing tools are obscure, hard to use, expensive, don’t actually provide much help, or all three.”

The SC Test category asked entrants “to design a tool, or set of tools, which will help programmers construct and maintain black box and glass box tests of software components at all levels, including functions, modules, and classes, and whole programs.”

The SC Test category is interesting in that the competition administrators clearly found it difficult to specify what they wanted to see in an entry. In fact, the whole category was reopened with a refined set of rules and expectations.

Ultimately, it’s difficult to tell whether this category made a significant difference. Where the tools to write tests used to be very sparse and difficult to use they are now many and several options exist for most programming languages. With this proliferation, several tried-and-tested methodologies have emerged which are consistent across many different tools, so while things still aren’t perfect they are much better.

In recent years there has been a culture shift in the wider software development community towards both testing in general and test-first development, where the tests for a new feature are written first, and then the implementation is coded incrementally until all tests pass.

The current challenge is to transfer this culture shift to the academic research community!

Tools for collaborative markdown editing

Thu, 15 Sep 2016 20:52:35 +0100

Photo by Alan Cleaver

I really love Markdown¹. I love its simplicity; its readability; its plain-text nature. I love that it can be written and read with nothing more complicated than a text-editor. I love how nicely it plays with version control systems. I love how easy it is to convert to different formats with Pandoc and how it’s become effectively the native text format for a wide range of blogging platforms.

One frustration I’ve had recently, then, is that it’s surprisingly difficult to collaborate on a Markdown document. There are various solutions that almost work but at best feel somehow inelegant, especially when compared with rock solid products like Google Docs. Finally, though, we’re starting to see some real possibilities. Here are some of the things I’ve tried, but I’d be keen to hear about other options.

1. Just suck it up

To be honest, Google Docs isn’t that bad. In fact it works really well, and has almost no learning curve for anyone who’s ever used Word (i.e. practically anyone who’s used a computer since the 90s). When I’m working with non-technical colleagues there’s nothing I’d rather use.

It still feels a bit uncomfortable though, especially the vendor lock-in. You can export a Google Doc to Word, ODT or PDF, but you need to use Google Docs to do that. Plus as soon as I start working in a word processor I get tempted to muck around with formatting.

2. Git(hub)

The obvious solution to most techies is to set up a GitHub repo, commit the document and go from there. This works very well for bigger documents written over a longer time, but seems a bit heavyweight for a simple one-page proposal, especially over short timescales.

Who wants to muck around with pull requests and merging changes for a document that’s going to take 2 days to write tops? This type of project doesn’t need a bug tracker or a wiki or a public homepage anyway. Even without GitHub in the equation, using git for such a trivial use case seems clunky.

3. Markdown in Etherpad/Google Docs

Etherpad is great tool for collaborative editing, but suffers from two key problems: no syntax highlighting or preview for markdown (it’s just treated as simple text); and you need to find a server to host it or do it yourself.

However, there’s nothing to stop you editing markdown with it. You can do the same thing in Google Docs, in fact, and I have. Editing a fundamentally plain-text format in a word processor just feels weird though.

4. Overleaf/Authorea

Overleaf and Authorea are two products developed to support academic editing. Authorea has built-in markdown support but lacks proper simultaneous editing. Overleaf has great simultaneous editing but only supports markdown by wrapping a bunch of LaTeX boilerplate around it. Both OK but unsatisfactory.

5. StackEdit

Now we’re starting to get somewhere. StackEdit has both Markdown syntax highlighting and near-realtime preview, as well as integrating with Google Drive and Dropbox for file synchronisation.

6. HackMD

HackMD is one that I only came across recently, but it looks like it does exactly what I’m after: a simple markdown-aware editor with live preview that also permits simultaneous editing. I’m a little circumspect simply because I know simultaneous editing is difficult to get right, but it certainly shows promise.

7. Classeur

I discovered Classeur literally today: it’s developed by the same team as StackEdit (which is now apparently no longer in development), and is currently in beta, but it looks to offer two killer features: real-time collaboration, including commenting, and pandoc-powered export to loads of different formats.

Anything else?

Those are the options I’ve come up with so far, but they can’t be the only ones. Is there anything I’ve missed?

Other plain-text formats are available. I’m also a big fan of org-mode. ↩︎

Software Carpentry: SC Track; hunt those bugs!

Mon, 12 Sep 2016 08:50:15 +0100

This competition will be an opportunity for the next wave of developers to show their skills to the world — and to companies like ours. — Dick Hardt, ActiveState (quote taken from SC Track page)

All code contains bugs, and all projects have features that users would like but which aren’t yet implemented. Open source projects tend to get more of these as their user communities grow and start requesting improvements to the product. As your open source project grows, it becomes harder and harder to keep track of and prioritise all of these potential chunks of work. What do you do?

The answer, as ever, is to make a to-do list. Different projects have used different solutions, including mailing lists, forums and wikis, but fairly quickly a whole separate class of software evolved: the bug tracker, which includes such well-known examples as Bugzilla, Redmine and the mighty JIRA.

Bug trackers are built entirely around such requests for improvement, and typically track them through workflow stages (planning, in progress, fixed, etc.) with scope for the community to discuss and add various bits of metadata. In this way, it becomes easier both to prioritise problems against each other and to use the hive mind to find solutions.

Unfortunately most bug trackers are big, complicated beasts, more suited to large projects with dozens of developers and hundreds or thousands of users. Clearly a project of this size is more difficult to manage and requires a certain feature set, but the result is that the average bug tracker is non-trivial to set up for a small single-developer project.

The SC Track category asked entrants to propose a better bug tracking system. In particular, the judges were looking for something easy to set up and configure without compromising on functionality.

The winning entry was a bug-tracker called Roundup, proposed by Ka-Ping Yee. Here we have another tool which is still in active use and development today. Given that there is now a huge range of options available in this area, including the mighty github, this is no small achievement.

These days, of course, github has become something of a de facto standard for open source project management. Although ostensibly a version control hosting platform, each github repository also comes with a built-in issue tracker, which is also well-integrated with the “pull request” workflow system that allows contributors to submit bug fixes and features themselves.

Github’s competitors, such as GitLab and Bitbucket, also include similar features. Not everyone wants to work in this way though, so it’s good to see that there is still a healthy ecosystem of open source bug trackers, and that Software Carpentry is still having an impact.

Software Carpentry: SC Config; write once, compile anywhere

Fri, 26 Aug 2016 19:47:40 +0100

Nine years ago, when I first release Python to the world, I distributed it with a Makefile for BSD Unix. The most frequent questions and suggestions I received in response to these early distributions were about building it on different Unix platforms. Someone pointed me to autoconf, which allowed me to create a configure script that figured out platform idiosyncracies Unfortunately, autoconf is painful to use – its grouping, quoting and commenting conventions don’t match those of the target language, which makes scripts hard to write and even harder to debug. I hope that this competition comes up with a better solution — it would make porting Python to new platforms a lot easier!
— Guido van Rossum, Technical Director, Python Consortium (quote taken from SC Config page)

On to the next Software Carpentry competition category, then. One of the challenges of writing open source software is that you have to make it run on a wide range of systems over which you have no control. You don’t know what operating system any given user might be using or what libraries they have installed, or even what versions of those libraries.

This means that whatever build system you use, you can’t just send the Makefile (or whatever) to someone else and expect everything to go off without a hitch. For a very long time, it’s been common practice for source packages to include a configure script that, when executed, runs a bunch of tests to see what it has to work with and sets up the Makefile accordingly. Writing these scripts by hand is a nightmare, so tools like autoconf and automake evolved to make things a little easier.

They did, and if the tests you want to use are already implemented they work very well indeed. Unfortunately they’re built on an unholy combination of shell scripting and the archaic Gnu M4 macro language. That means if you want to write new tests you need to understand both of these as well as the architecture of the tools themselves — not an easy task for the average self-taught research programmer.

SC Conf, then, called for a re-engineering of the autoconf concept, to make it easier for researchers to make their code available in a portable, platform-independent format. The second round configuration tool winner was SapCat, “a tool to help make software portable”. Unfortunately, this one seems not to have gone anywhere, and I could only find the original proposal on the Internet Archive.

There were a lot of good ideas in this category about making catalogues and databases of system quirks to avoid having to rerun the same expensive tests again the way a standard ./configure script does. I think one reason none of these ideas survived is that they were overly ambitions, imagining a grand architecture where their tool provide some overarching source of truth. This is in stark contrast to the way most Unix-like systems work, where each tool does one very specific job well and tools are easy to combine in various ways.

In the end though, I think Moore’s Law won out here, making it easier to do the brute-force checks each time than to try anything clever to save time — a good example of avoiding unnecessary optimisation. Add to that the evolution of the generic pkg-config tool from earlier package-specific tools like gtk-config, and it’s now much easier to check for particular versions and features of common packages.

On top of that, much of the day-to-day coding of a modern researcher happens in interpreted languages like Python and R, which give you a fully-functioning pre-configured environment with a lot less compiling to do.

As a side note, Tom Tromey, another of the shortlisted entrants in this category, is still a major contributor to the open source world. He still seems to be involved in the automake project, contributes a lot of code to the emacs community too and blogs sporadically at The Cliffs of Inanity.

Semantic linefeeds: one clause per line

Mon, 22 Aug 2016 21:05:45 +0100

I’ve started using “semantic linefeeds”, a concept I discovered on Brandon Rhodes’ blog, when writing content, an idea described in that article far better than I could. I turns out this is a very old idea, promoted way back in the day by Brian W Kernighan, contributor to the original Unix system, co-creator of the AWK and AMPL programming languages and co-author of a lot of seminal programming textbooks including “The C Programming Language”.

The basic idea is that you break lines at natural gaps between clauses and phrases, rather than simply after the last word before you hit 80 characters. Keeping line lengths strictly to 80 characters isn’t really necessary in these days of wide aspect ratios for screens. Breaking lines at points that make semantic sense in the sentence is really helpful for editing, especially in the context of version control, because it isolates changes to the clause in which they occur rather than just the nearest 80-character block.

I also like it because it makes my crappy prose feel just a little bit more like poetry. ☺

Software Carpentry: SC Build; or making a better make

Fri, 19 Aug 2016 19:30:13 +0100

Software tools often grow incrementally from small beginnings into elaborate artefacts. Each increment makes sense, but the final edifice is a mess. make is an excellent example: a simple tool that has grown into a complex domain-specific programming language. I look forward to seeing the improvements we will get from designing the tool afresh, as a whole…
— Simon Peyton-Jones, Microsoft Research (quote taken from SC Build page)

Most people who have had to compile an existing software tool will have come across the venerable make tool (which usually these days means GNU Make). It allows the developer to write a declarative set of rules specifying how the final software should be built from its component parts, mostly source code, allowing the build itself to be carried out by simply typing make at the command line and hitting Enter.

Given a set of rules, make will work out all the dependencies between components and ensure everything is built in the right order and nothing that is up-to-date is rebuilt. Great in principle but make is notoriously difficult for beginners to learn, as much of the logic for how builds are actually carried out is hidden beneath the surface. This also makes it difficult to debug problems when building large projects. For these reasons, the SC Build category called for a replacement build tool engineered from the ground up to solve these problems.

The second round winner, ScCons, is a Python-based make-like build tool written by Steven Knight. While I could find no evidence of any of the other shortlisted entries, this project (now renamed SCons) continues in active use and development to this day.

I actually use this one myself from time to time and to be honest I prefer it in many cases to trendy new tools like rake or grunt and the behemoth that is Apache Ant. Its Python-based SConstruct file syntax is remarkably intuitive and scales nicely from very simple builds up to big and complicated project, with good dependency tracking to avoid unnecessary recompiling. It has a lot of built-in rules for performing common build & compile tasks, but it’s trivial to add your own, either by combining existing building blocks or by writing a new builder with the full power of Python.

A minimal SConstruct file looks like this:

Program('hello.c')

Couldn’t be simpler! And you have the full power of Python syntax to keep your build file simple and readable.

It’s interesting that all the entries in this category apart from one chose to use a Python-derived syntax for describing build steps. Python was clearly already a language of choice for flexible multi-purpose computing. The exception is the entry that chose to use XML instead, which I think is a horrible idea (oh how I used to love XML!) but has been used to great effect in the Java world by tools like Ant and Maven.

What happened to the original Software Carpentry?

Tue, 16 Aug 2016 18:03:13 +0100

“Software Carpentry was originally a competition to design new software tools, not a training course. The fact that you didn’t know that tells you how well it worked.”

When I read this in a recent post on Greg Wilson’s blog, I took it as a challenge. I actually do remember the competition, although looking at the dates it was long over by the time I found it.

I believe it did have impact; in fact, I still occasionally use one of the tools it produced, so Greg’s comment got me thinking: what happened to the other competition entries?

Working out what happened will need a bit of digging, as most of the relevant information is now only available on the Internet Archive. It certainly seems that by November 2008 the domain name had been allowed to lapse and had been replaced with a holding page by the registrar.

There were four categories in the competition, each representing a category of tool that the organisers thought could be improved:

SC Build: a build tool to replace make
SC Conf: a configuration management tool to replace autoconf and automake
SC Track: a bug tracking tool
SC Test: an easy to use testing framework

I’m hoping to be able to show that this work had a lot more impact than Greg is admitting here. I’ll keep you posted on what I find!

Changing static site generators: Nanoc → Hugo

Fri, 12 Aug 2016 13:18:28 +0100

I’ve decided to move the site over to a different static site generator, Hugo. I’ve been using Nanoc for a long time and it’s worked very well, but lately it’s been taking longer and longer to compile the site and throwing weird errors that I can’t get to the bottom of.

At the time I started using Nanoc, static site generators were in their infancy. There weren’t the huge number of feature-loaded options that there are now, so I chose one and I built a whole load of blogging-related functionality myself. I did it in ways that made sense at the time but no longer work well with Nanoc’s latest versions.

So it’s time to move to something that has blogging baked-in from the beginning and I’m taking the opportunity to overhaul the look and feel too. Again, when I started there weren’t many pre-existing themes so I built the whole thing myself and though I’m happy with the work I did on it it never quite felt polished enough. Now I’ve got the opportunity to adapt one of the many well-designed themes already out there, so I’ve taken one from the Hugo themes gallery and tweaked the colours to my satisfaction.

Hugo also has various features that I’ve wanted to implement in Nanoc but never quite got round to it. The nicest one is proper handling of draft posts and future dates, but I keep finding others.

There’s a lot of old content that isn’t quite compatible with the way Hugo does things so I’ve taken the old Nanoc-compiled content and frozen it to make sure that old links should still work.

I could probably fiddle with it for years without doing much so it’s probably time to go ahead and publish it. I’m still not completely happy with my choice of theme but one of the joys of Hugo is that I can change that whenever I want.

Let me know what you think!

License

Wed, 10 Aug 2016 08:37:18 +0100

Except where otherwise stated, all content on eRambler by Jez Cope is licensed under a Creative Commons Attribution-ShareAlike 4.0 International license.

RDM Resources

Tue, 19 Jul 2016 07:42:33 +0100

I occasionally get asked for resources to help someone learn more about research data management (RDM) as a discipline (i.e. for those providing RDM support rather than simply wanting to manage their own data). I’ve therefore collected a few resources together on this page. If you’re lucky I might even update it from time to time!

First, a caveat: this is very focussed on UK Higher Education, though much of it will still be relevant for people outside that narrow demographic.

My general recommendation would be to start with the Digital Curation Centre (DCC) website and follow links out from there. I also have a slowly growing list of RDM links on Diigo, and there’s an RDM section in my list of blogs and feeds too.

Mailing lists

Jiscmail is a popular list server run for the benefit of further and higher education in the UK; the following lists are particularly relevant:
- RESEARCH-DATAMAN
- DATA-PUBLICATION
- DIGITAL-PRESERVATION
- LIS-RESEARCHSUPPORT
The Research Data Alliance have a number of Interest Groups and Working Groups that discuss issues by email

Events

International Digital Curation Conference — major annual conference
Research Data Management Forum — roughly every six months, places are limited!
RDA Plenary — also every 6 months, but only about 1 in every 3 in Europe

Books

In no particular order:

Martin, Victoria. Demystifying eResearch: A Primer for Librarians. Libraries Unlimited, 2014.
Borgman, Christine L. Big Data, Little Data, No Data: Scholarship in the Networked World. Cambridge, Massachusetts: The MIT Press, 2015.
Corti, Louise, Veerle Van den Eynden, and Libby Bishop. Managing and Sharing Research Data. Thousand Oaks, CA: SAGE Publications Ltd, 2014.
Pryor, Graham, ed. Managing Research Data. Facet Publishing, 2012.
Pryor, Graham, Sarah Jones, and Angus Whyte, eds. Delivering Research Data Management Services: Fundamentals of Good Practice. Facet Publishing, 2013.
Ray, Joyce M., ed. Research Data Management: Practical Strategies for Information Professionals. West Lafayette, Indiana: Purdue University Press, 2014.

Reports

‘Ten Recommendations for Libraries to Get Started with Research Data Management’. LIBER, 24 August 2012. http://libereurope.eu/news/ten-recommendations-for-libraries-to-get-started-with-research-data-management/.
‘Science as an Open Enterprise’. Royal Society, 2 June 2012. https://royalsociety.org/policy/projects/science-public-enterprise/Report/.
Mary Auckland. ‘Re-Skilling for Research’. RLUK, January 2012. http://www.rluk.ac.uk/wp-content/uploads/2014/02/RLUK-Re-skilling.pdf.

Journals

International Journal of Digital Curation (IJDC)
Journal of eScience Librarianship (JeSLib)

Fairphone 2: initial thoughts on the original ethical smartphone

Sat, 07 May 2016 16:56:29 +0100

I’ve had my eye on the Fairphone 2 for a while now, and when my current phone, an aging Samsung Galaxy S4, started playing up I decided it was time to take the plunge. A few people have asked for my thoughts on the Fairphone so here are a few notes.

Why I bought it

The thing that sparked my interest, and the main reason for buying the phone really, was the ethical stance of the manufacturer. The small Swedish company have gone to great lengths to ensure that both labour and materials are sourced as responsibly as possible. They regularly inspect the factories where the parts are made and assembled to ensure fair treatment of the workers and they source all the raw materials carefully to minimise the environmental impact and the use of conflict minerals.

Another side to this ethical stance is a focus on longevity of the phone itself. This is not a product with an intentionally limited lifespan. Instead, it’s designed to be modular and as repairable as possible, by the owner themselves. Spares are available for all of the parts that commonly fail in phones (including screen and camera), and at the time of writing the Fairphone 2 is the only phone to receive 10/10 for reparability from iFixit. There are plans to allow hardware upgrades, including an expansion port on the back so that NFC or wireless charging could be added with a new case, for example.

What I like

So far, the killer feature for me is the dual SIM card slots. I have both a personal and a work phone, and the latter was always getting left at home or in the office or running out of charge. Now I have both SIMs in the one phone: I can recieve calls on either number, turn them on and off independently and choose which account to use when sending a text or making a call.

The OS is very close to “standard” Android, which is nice, and I really don’t miss all the extra bloatware that came with the Galaxy S4. It also has twice the storage of that phone, which is hardly unique but is still nice to have.

Overall, it seems like a solid, reliable phone, though it’s not going to outperform anything else at the same price point. It certainly feels nice and snappy for everything I want to use it for. I’m no mobile gamer, but there is that distant promise of upgradability on the horizon if you are.

What I don’t like

I only have two bugbears so far. Once or twice it’s locked up and become unresponsive, requiring a “manual reset” (removing and replacing the battery) to get going again. It also lacks NFC, which isn’t really a deal breaker, but I was just starting to make occasional use of it on the S4 (mostly experimenting with my Yubikey NEO) and it would have been nice to try out Android Pay when it finally arrives in the UK.

Overall

It’s definitely a serious contender if you’re looking for a new smartphone and aren’t bothered about serious mobile gaming. You do pay a premium for the ethical sourcing and modularity, but I feel that’s worth it for me. I’m looking forward to seeing how it works out as a phone.

Wiring my web

Fri, 01 Apr 2016 17:37:00 +0100

I’m a nut for automating repetitive tasks, so I was dead pleased a few years ago when I discovered that IFTTT let me plug different bits of the web together. I now use it for tasks such as:

Syndicating blog posts to social media
Creating scheduled/repeating todo items from a Google Calendar
Making a note to revisit an article I’ve starred in Feedly

I’d probably only be half-joking if I said that I spend more time automating things than I save not having to do said things manually. Thankfully it’s also a great opportunity to learn, and recently I’ve been thinking about reimplementing some of my IFTTT workflows myself to get to grips with how it all works.

There are some interesting open source projects designed to offer a lot of this functionality, such as Huginn, but I decided to go for a simpler option for two reasons:

I want to spend my time learning about the APIs of the services I use and how to wire them together, rather than learning how to use another big framework; and
I only have a small Amazon EC2 server to pay with and a heavy Ruby on Rails app like Huginn (plus web server) needs more memory than I have.

Instead I’ve gone old-school with a little collection of individual scripts to do particular jobs. I’m using the built-in scheduling functionality of systemd, which is already part of a modern Linux operating system, to get them to run periodically. It also means I can vary the language I use to write each one depending on the needs of the job at hand and what I want to learn/feel like at the time. Currently it’s all done in Python, but I want to have a go at Lisp sometime, and there are some interesting new languages like Go and Julia that I’d like to get my teeth into as well.

You can see my code on github as it develops: https://github.com/jezcope/web-plumbing. Comments and contributions are welcome (if not expected) and let me know if you find any of the code useful.

Image credit: xkcd #1319, Automation

Data is like water, and language is like clothing

Thu, 31 Mar 2016 17:40:00 +0100

I admit it: I’m a grammar nerd. I know the difference between ‘who’ and ‘whom’, and I’m proud.

I used to be pretty militant, but these days I’m more relaxed. I still take joy in the mechanics of the language, but I also believe that English is defined by its usage, not by a set of arbitrary rules. I’m just as happy to abuse it as to use it, although I still think it’s important to know what rules you’re breaking and why.

My approach now boils down to this: language is like clothing. You (probably) wouldn’t show up to a job interview in your pyjamas¹, but neither are you going to wear a tuxedo or ballgown to the pub.

Getting commas and semicolons in the right place is like getting your shirt buttons done up right. Getting it wrong doesn’t mean you’re an idiot. Everyone will know what you meant. It will affect how you’re perceived, though, and that will affect how your message is perceived.

And there are former rules² that some still enforce that are nonetheless dropping out of regular usage. There was a time when everyone in an office job wore formal clothing. Then it became acceptable just to have a blouse, or a shirt and tie. Then the tie became optional and now there are many professions where perfectly well-respected and competent people are expected to show up wearing nothing smarter than jeans and a t-shirt.

One such rule IMHO is that ‘data’ is a plural and should take pronouns like ’they’ and ’these’. The origin of the word ‘data’ is in the Latin plural of ‘datum’, and that idea has clung on for a considerable period. But we don’t speak Latin and the English language continues to evolve: ‘agenda’ also began life as a Latin plural, but we don’t use the word ‘agendum’ any more. It’s common everyday usage to refer to data with singular pronouns like ‘it’ and ’this’, and it’s very rare to see someone referring to a single datum (as opposed to ‘data point’ or something).

If you want to get technical, I tend to think of data as a mass noun, like ‘water’ or ‘information’. It’s uncountable: talking about ‘a water’ or ‘an information’ doesn’t make much sense, but it uses singular pronouns, as in ’this information’. If you’re interested, the Oxford English Dictionary also takes this position, while Chambers leaves the choice of singular or plural noun up to you.

There is absolutely nothing wrong, in my book, with referring to data in the plural as many people still do. But it’s no longer a rule and for me it’s weakened further from guideline to preference.

It’s like wearing a bow-tie to work. There’s nothing wrong with it and some people really make it work, but it’s increasingly outdated and even a little eccentric.

or maybe you’d totally rock it. ↩︎
Like not starting a sentence with a conjunction… ↩︎

#IDCC16 day 2: new ideas

Wed, 16 Mar 2016 07:44:14 +0100

Well, I did a great job of blogging the conference for a couple of days, but then I was hit by the bug that’s been going round and didn’t have a lot of energy for anything other than paying attention and making notes during the day! I’ve now got round to reviewing my notes so here are a few reflections on day 2.

Day 2 was the day of many parallel talks! So many great and inspiring ideas to take in! Here are a few of my take-home points.

Big science and the long tail

The first parallel session had examples of practical data management in the real world. Jian Qin & Brian Dobreski (School of Information Studies, Syracuse University) worked on reproducibility with one of the research groups involved with the recent gravitational wave discovery. “Reproducibility” for this work (as with much of physics) mostly equates to computational reproducibility: tracking the provenance of the code and its input and output is key. They also found that in practice the scientists’ focus was on making the big discovery, and ensuring reproducibility was seen as secondary. This goes some way to explaining why current workflows and tools don’t really capture enough metadata.

Milena Golshan & Ashley Sands (Center for Knowledge Infrastructures, UCLA) investigated the use of Software-as-a-Service (SaaS, such as Google Drive, Dropbox or more specialised tools) as a way of meeting the needs of long-tail science research such as ocean science. This research is characterised by small teams, diverse data, dynamic local development of tools, local practices and difficulty disseminating data. This results in a need for researchers to be generalists, as opposed to “big science” research areas, where they can afford to specialise much more deeply. Such generalists tend to develop their own isolated workflows, which can differ greatly even within a single lab. Long-tail research also often struggles from a lack of dedicated IT support. They found that use of SaaS could help to meet these challenges, but with a high cost required to cover the needed guarantees of security and stability.

Education & training

This session focussed on the professional development of library staff. Eleanor Mattern (University of Pittsburgh) described the immersive training introduced to improve librarians’ understanding of the data needs of their subject areas in delivering their RDM service delivery model. The participants each conducted a “disciplinary deep dive”, shadowing researchers and then reporting back to the group on their discoveries with a presentation and discussion.

Liz Lyon (also University of Pittsburgh, formerly UKOLN/DCC) gave a systematic breakdown of the skills, knowledge and experience required in different data-related roles, obtained from an analysis of job adverts. She identified distinct roles of data analyst, data engineer and data journalist, and as well as each role’s distinctive skills, pinpointed common requirements of all three: Python, R, SQL and Excel. This work follows on from an earlier phase which identified an allied set of roles: data archivist, data librarian and data steward.

This session gave an overview of several specific workflow tools designed for researchers. Marisa Strong (University of California Curation Centre/California Digital Libraries) presented Dash, a highly modular tool for manual data curation and deposit by researchers. It’s built on their flexible backend, Stash, and though it’s currently optimised to deposit in their Merritt data repository it could easily be hooked up to other repositories. It captures DataCite metadata and a few other fields, and is integrated with ORCID to uniquely identify people.

In a different vein, Eleni Castro (Institute for Quantitative Social Science, Harvard University) discussed some of the ways that Harvard’s Dataverse repository is streamlining deposit by enabling automation. It provides a number of standardised endpoints such as OAI-PMH for metadata harvest and SWORD for deposit, as well as custom APIs for discovery and deposit. Interesting use cases include:

An addon for the Open Science Framework to deposit in Dataverse via SWORD
An R package to enable automatic deposit of simulation and analysis results
Integration with publisher workflows Open Journal Systems
A growing set of visualisations for deposited data

In the future they’re also looking to integrate with DMPtool to capture data management plans and with Archivematica for digital preservation.

Andrew Treloar (Australian National Data Service) gave us some reflections on the ANDS “applications programme”, a series of 25 small funded projects intended to address the fourth of their strategic transformations, single use → reusable. He observed that essentially these projects worked because they were able to throw money at a problem until they found a solution: not very sustainable. Some of them stuck to a traditional “waterfall” approach to project management, resulting in “the right solution 2 years late”. Every researcher’s needs are “special” and communities are still constrained by old ways of working. The conclusions from this programme were that:

“Good enough” is fine most of the time
Adopt/Adapt/Augment is better than Build
Existing toolkits let you focus on the 10% functionality that’s missing
Succussful projects involved research champions who can: 1) articulate their community’s requirements; and 2) promote project outcomes

Summary

All in all, it was a really exciting conference, and I’ve come home with loads of new ideas and plans to develop our services at Sheffield. I noticed a continuation of some of the trends I spotted at last year’s IDCC, especially an increasing focus on “second-order” problems: we’re no longer spending most of our energy just convincing researchers to take data management seriously and are able to spend more time helping them to do it better and get value out of it. There’s also a shift in emphasis (identified by closing speaker Cliff Lynch) from sharing to reuse, and making sure that data is not just available but valuable.

#IDCC16 Day 1: Open Data

Tue, 23 Feb 2016 19:43:57 +0100

The main conference opened today with an inspiring keynote by Barend Mons, Professor in Biosemantics, Leiden University Medical Center. The talk had plenty of great stuff, but two points stood out for me.

First, Prof Mons described a newly discovered link between Huntingdon’s Disease and a previously unconsidered gene. No-one had previously recognised this link, but on mining the literature, an indirect link was identified in more than 10% of the roughly 1 million scientific claims analysed. This is knowledge for which we already had more than enough evidence, but which could never have been discovered without such a wide-ranging computational study.

Second, he described a number of behaviours which should be considered “malpractice” in science:

Relying on supplementary data in articles for data sharing: the majority of this is trash (paywalled, embedded in bitmap images, missing)
Using the Journal Impact Factor to evaluate science and ignoring altmetrics
Not writing data stewardship plans for projects (he prefers this term to “data management plan”)
Obstructing tenure for data experts by assuming that all highly-skilled scientists must have a long publication record

A second plenary talk from Andrew Sallons of the Centre for Open Science introduced a number of interesting-looking bits and bobs, including the Transparency & Openness Promotion (TOP) Guidelines which set out a pathway to help funders, publishers and institutions move towards more open science.

The rest of the day was taken up with a panel on open data, a poster session, some demos and a birds-of-a-feather session on sharing sensitive/confidential data. There was a great range of posters, but a few that stood out to me were:

Lessons learned about ISO 16363 (“Audit and certification of trustworthy digital repositories”) certification from the British Library
Two separate posters (from the Universities of Toronto and Colorado) about disciplinary RDM information & training for liaison librarians
A template for sharing psychology data developed by a psychologist-turned-information researcher from Carnegie Mellon University

More to follow, but for now it’s time for the conference dinner!

#IDCC16 Day 0: business models for research data management

Mon, 22 Feb 2016 18:20:55 +0100

I’m at the International Digital Curation Conference 2016 (#IDCC16) in Amsterdam this week. It’s always a good opportunity to pick up some new ideas and catch up with colleagues from around the world, and I always come back full of new possibilities. I’ll try and do some more reflective posts after the conference but I thought I’d do some quick reactions while everything is still fresh.

Monday and Thursday are pre- and post-conference workshop days, and today I attended Developing Research Data Management Services. Joy Davidson and Jonathan Rans from the Digital Curation Centre (DCC) introduced us to the Business Model Canvas, a template for designing a business model on a single sheet of paper. The model prompts you to think about all of the key facets of a sustainable, profitable business, and can easily be adapted to the task of building a service model within a larger institution. The DCC used it as part of the Collaboration to Clarify Curation Costs (4C) project, whose output the Curation Costs Exchange is also worth a look.

It was a really useful exercise to be able to work through the whole process for an aspect of research data management (my table focused on training & guidance provision), both because of the ideas that came up and also the experience of putting the framework into practice. It seems like a really valuable tool and I look forward to seeing how it might help us with our RDM service development.

Tomorrow the conference proper begins, with a range of keynotes, panel sessions and birds-of-a-feather meetings so hopefully more then!

About me

Mon, 01 Jan 0001 00:00:00 +0000

I help researchers communicate and collaborate more effectively using technology, mainly focusing on research data management policy, practice, training and advocacy. I currently work at The British Library as Data Services Lead.

In my free time, I like to:

Run, hike, climb
Play and run tabletop roleplaying games (Dungeons & Dragons 5e, Numenera, Cypher System, Tales from the Loop) and board games (including Go)
Play the accordion, dance British & European social dances, morris dance
Cook, write, read (fiction and non-fiction, mostly scifi & fantasy)

Professional service & recognition

Member, CAKE-CoSeC EDI Working Group, 2025 – present
Editorial Board Member, Journal of Open Humanities Data (JOHD), 2023 – present
Member and former chair, EMEA Expert Group, 2021 – present (chair until 2023)
Fellow, Sustainable Software Institute, 2020 inauguration (see also my Fellow profile)
Member, DataCite Community Engagement Steering Group, 2018 – present
LIBER Award for Library Innovation (shared), for: Chue Hong, N. et al. (2021) “Recognising the value of software: how libraries can help the adoption of software citation,” in Liber annual conference 2021. LIBER. Available at: https://doi.org/10.6084/m9.figshare.14825268.
Open Life Science programme Mentor & Expert, 2020 – present

Formerly

Committee Member (Website), Neurodivergent Library and Information Staff Network (NLISN), 2024 – 2025
Member, Physical Sciences Data Science Services (PSDS) Management Advisory Panel, 2019 – 2023
SSI Fellowship mentor, 2019
Member, Digital Preservation Coalition (DPC) Workforce Development Subcommittee, 2018 – 2023
Member, Library Carpentry Governance Group, 2017 – 2019
Maintainer of the Library Carpentry lesson about git
Member of the Jisc UK Research Data Shared Service Expert Advisory Group

Projects

Mon, 01 Jan 0001 00:00:00 +0000

I'm trying to move away from relying on any one forge (e.g. GitHub, GitLab, etc.) for project hosting, so here's a rundown of some of the little open source projects I have scattered around the web right now.

Some of them have their own issue trackers — feel free to tell me about bugs or make pull requests there — but you can also send me bugs, comments or patches at projects@erambler.co.uk.

Active

Projects that can be considered current: I may or may not have made any significant changes to them recently, but I certainly use them and I would love it if you did too!

mxadm

A simple CLI interface to basic Matrix room admin tasks.

Source: https://tildegit.org/petrichor/mxadm
Rust crate: mxadm on lib.rs; mxadm on crates.io

Template iPython magics 🎩

This package provides simple IPython magics to render cells as templates in a variety of different templating languages. It currently supports Mako and Jinja2.

Source: https://gitlab.com/jezcope/template-ipython-magic
Python package: template-ipython-magic on PyPI

remarkable-scripts

A small collection of useful scripts for users of reMarkable paper tablets.

Source: https://tildegit.org/petrichor/remarkable-scripts

Dormant or nascent

Projects that I've started and done some work on, often to the point of a working prototype, or that are just ideas so far, and that I intend to develop further at some point.

pyrefine

Execute OpenRefine JSON scripts without OpenRefine (or Java).

Source: https://github.com/jezcope/pyrefine

csvy-python

Read CSVY data packages from Python.

Source: https://github.com/jezcope/csvy-python

nix-gently

Experiments & toys

Things that were done to try something out, or to do a very simple job that I no longer need.

doi2oa
altmedals2012
runkeeper-data
Shipton Mill queue checker

Educational examples

Things that simply exist to demonstrate for others how a thing can be done.

binder-nix-flakes: Demo of using the "flakes" feature of NixOS and it's cross-platform package manager, nix, to specify the dependencies for a Binder of Jupyter notebooks. https://tildegit.org/petrichor/binder-nix-flakes
zenodo-example-py: Demo of accessing the Zenodo REST API from Python to fetch records. https://gitlab.com/jezcope/zenodo-example-py

Abandoned

Stuff that I started and used for a while, but which no longer serve any purpose for me. Feel free to learn from these or fork them as the basis for something of your own

sagadoc: A really simple tool to generate documents from data, code and templates. Heavily inspired by Org mode for Emacs. I mostly use org-mode or Jupyter Notebooks for this these days. https://github.com/jezcope/sagadoc
dropbox-conflicts-el: An emacs package to detect conflicted copies of files in your Dropbox on open. https://github.com/jezcope/dropbox-conflicts-el