IA Summit Logo

Summit 2014 March 26-30 in San Diego

The Path Ahead Lovely San Diego

All you need to know about our community review process

In the 15th year of the IA Summit, we received the most proposals we have ever seen: 416! Luckily, 100 people have volunteered their time and will blind review around 15 proposals each. In line with our commitment to transparency, we’ll cover the Frequently Asked Questions about our process in this post.

What’s the overall review process?

1) Every proposal will be blind reviewed by at least two, ideally three different people.
2) Once everybody has provided their feedback, our 6 curators will read through all the proposals, taking into consideration each talk’s blind review feedback, connection to our theme, and the reputation of the speaker. Curators work in pairs, and each pair works on one of our three conference tracks.
3) The curation team passes their recommendations to the conference chairs. The chairs then finalise the schedule. Around 60 sessions out of the 416 submissions will end up forming the program.

Abby has made a wonderful diagram that illustrates the entire curation process. Contains data!

How a proposal comes to life at #IAS14

How a proposal comes to life at #IAS14

Why the whole blind review business?

Rather than conferences organised by a company that are curated solely by the organisers, the IA Summit uses blind reviews, a process common at academic conferences. We value this process as it involves the community, and gives people ownership over the event.

Reviewing a proposal ‘blind’, i.e. without knowing the identity of the submitter, allows reviewers to focus on the quality and appeal of the proposal.

This feedback is invaluable for the curation team and conference chairs, as it helps to

- assess the clarity and fit of a proposal
- understand what topics and themes emerge
- get more than one brain’s opinion!

The blind review feedback on the quality of content also helps curators and chairs to resolve conflicts, e.g. if two people have submitted a similar topic and could both deliver it well, it’s the best content that wins.

So, do you just pick the top sessions that were best rated in the blind review?

No. The blind review feedback informs the curation team’s decisions. They take a look at every single proposal, knowing who it was submitted by, and consider the feedback, the fit with the theme, and the speaker’s identity to curate their track. After the curators propose which sessions to consider for the program, the chairs make the final decision about the schedule.

How do you decide who is in?

We consider:
Content + ability of speaker to deliver on this topic + conference theme and curator vision

The most important criteria is the quality of the proposal’s content. Secondly, we’ll consider if the submitter is the best person to deliver this content. Thirdly, the curators take their vision for their track into account.

The curators and chairs will strive for a balanced program, with first timers and experienced speakers, with innies, consultants, and freelancers, with speakers from different contexts and locations, etc. Again, content quality is crucial. We don’t have a first-time speaker or male/female or people from outside the US quota we are trying to hit. But if we have to choose between two awesome sessions and one has been proposed by a person who would offer a different perspective due to who they are, we’ll go with theirs.

How do you make sure the reviewers give useful feedback?

Apart from asking submitters for more details in a more structured submission form, we are coaching all our reviewers on how to provide useful feedback. This is reflected in the instructions and review form, and it’s all based on Aaron’s expertise on the topic.

Here is Aaron’s video on how to provide great feedback, that we shared with our reviewers.

http://vimeo.com/76078850 (Password: IASummit2014)

Why is the review form so ugly?

We are using a system called START that is super-helpful for the review process. When we had a look around for alternatives, we couldn’t find a more usable system that was within our budget and timeframe. Amongst other things, START allows you to allocate proposals randomly to reviewers, manage all of our communications to different groups, and it even has a nifty feature that helps to chase reviewers. It even identifies potential reviewer conflicts such as working at a company with the person you are reviewing the content of.

…and if you’re thinking, why not use Google Docs, previous chairs have tried it and we heard it wasn’t fun.

Remind me, what’s the theme again?

Read up on this year’s theme, The Path Ahead, here. On this page, you’ll also find the questions we blogged as an inspiration for submitters.

I have more questions! Who do I contact?

Please email IAS14-curation@asis.org and either Sandra, our curation manager, or Johanna, one of the chairs, will get back to you.