Friday, March 26, 2010

Assessing open source software as a scholarly contribution

A post that is not strictly related to Computer Graphics, 3D or Cultural Heritage.
Just a small note/rant to point out a recent paper:

Lou Hafer and Arthur E. Kirkpatrick
"Assessing open source software as a scholarly contribution"
Communications of the ACM, Volume 52 ,  Issue 12  (December 2009)

It is an interesting discussion on the fact that "Academic computer science has an odd relationship with software: Publishing papers about software is considered a distinctly stronger contribution than publishing the software".
Being a senior researcher before being the lead developer of MeshLab, I have to say that I totally agree with those feelings. I have often thought that devoting a significant portion of my time to the MeshLab project is not a 100% wise move from a career point of view; probably writing a bunch of easy minor-variation papers is much more rewarding and is evaluated better when running for higher positions.

The sad thing is that there are people thinking that the citations coming from the paper you have written about your software are more than enough to reward you for your effort of writing it. Usually these considerations came from computer scientists who do not have perfectly clear what means writing and maintaining real software tools.
Some bare facts:
  1. If you write and maintain significant software tools/library then you are serving the research community in a way that is more significant that writing a paper.
  2. The time required to develop and maintain sw tools is much larger than the time required to write one paper.
  3. Assessing the importance/significance of software is more difficult than assessing the value of papers, no common bibliometric tools (obviously download count is not a good metric).
  4. Commissions evaluating people careers usually ignore sw and concentrate on other, more standard, research products (papers, editorial boards, commitee, teaching, prizes, etc).
As a simple consequence, of 2,3,4 and despite of 1, with current career evaluation habits, developing and maintaining sw tools that are significantly useful for the research community is NOT a career maximizing move. And this is, in my humble opinion, definitely, completely, utterly WRONG.

Now when you stumble upon a discontinued piece of code that you would have loved to have maintained, you have an hint of why the original author abandoned it.

1 comment:

ErkDemon said...

Hello MeshLab People! :)

For getting scholarly credit, you really need the thing to be part of some sort of standardised index.

How about making use of the existing ISBN system? I //think// that you can still use ISBNs for software... Buying a block of ten ISBN numbers probably costs about GBP100-00. You have to decide on a unique publisher name, lodge your publisher details, and then assign ISBN numbers to your products (say, give each serious revision its own ISBN). Start citing the ISBN/publisher/etc in your papers, and other people who read them'll start doing the same, and then you can track the resulting citations normally through the citation indexes.

Th ISBN system uses different numbers for different editions/revisions, so I don't know whether you'd give 1.x its own code and wait until 2.x for the next, or whether you'd assign codes to 1.0, 1.1, 1.2, 1.3 ... if you feel that the sub-versions represent significant work, then the second approach gives you more "publications", but also dilutes your citations (and means that you run out of ISBNs faster).

Once the product has a code, all sorts of good things can happen. You can embed ISBN-ised links on your blog using COinS tags, so that the ISBN and associated details can be harvested as a reference by anyone usign Zotero, it'll show up on all the book indexes with you listed as author/designer/whatever, it can be listed on as an entity associated with you, and so on.

It might be worth considering.