Why OpenTranslators(TM) are important.

We found this post on the “Federated Search Blog” about our recent announcement at ALA concerning OpenTranslators. It raises some very important questions, conveys a misunderstanding and possibly some philosophical differences, but the important point is that it provides an opportunity to discuss why we believe OpenTranslators are important.

CARE Affiliates(TM) believes, as does the writer of the Blog above, that the OpenTranslators amounts to something “That… is pretty significant” in the federated searching world. What we did, was to bring together a total of three companies, each contributing expertise, products and ideas to create a new way for users to have access to thousands and thousands of databases, some of which support proprietary API’s of their own, but most of which support no interface at all beyond the end-user HTML interface. These have to be parsed and interpreted to support metasearching. Some represent freely available content, but most are subscription databases of the sort used in the hundreds by a typical library to meet patron needs. So bottom line, what does that mean? It means that the search interface used by your library and thus, your users, is yet further decoupled from the content underneath.

First, a little history. Years ago (and it is still true in some cases) if you wanted to search certain content, you could only do it using the interface provided by the company that assembled that content. The interface and content were tightly coupled. This was very frustrating for users as it meant that for each resource they wanted to use, they likely had to learn a new search interface.

Federated search tools eliminated that problem and allowed a common search interface layer to many resources. This was better for end users who now only had to learn one search interface; that provided by their federated search vendor. The downside was that in order to search to all those content resources, there had to be a translator or connector developed that could talk the language of the original content providers search interface and translate that to the language of the new federated search providers interface. So in the end, the vendors’ translators/connectors were tightly coupled to the vendors’ search interface. The search interface had moved up a layer in the stack, and you as the customer had more options/choices, but we saw the possibility to provide even more choices.

Our colleagues at Index Data(TM) had built software to take SRU/SRW/Z39.50 searches and translate them for use with translators/connectors. Couple that with the fact that we had signed a deal with WebFeat(TM) in August of 2007 that allowed us to sell their translators with our products. We did (and do) this with the metasearch/federated search application called MasterKey.

While we thought it was very progressive of WebFeat to allow us to couple an OSS based metasearch/federated search solution to their translators, all three companies had a vision of carrying the concept yet further. Why not take the Index Data SRU/SRW/Z39.50 translators, couple them with WebFeat’s translators and allow the end user to pick the metasearch interface of their choice? It could be their existing OPAC. It could be anything that could talk SRU/SRW/Z39.50; a large and growing selection of software products, both open and proprietary; it could even include homegrown applications and mashups. It would mean that we had yet further decoupled content from the search interface and were putting total control of the interface selected into the hands of the customers. That was the basis of the OpenTranslators announcement.

Now, with that history and background, let’s answer some of the questions/points raised in the Blog post that started this conversation.

First, we want to offer a correction to the statement that “WebFeat has a large number of translators (connectors to SRU/SRW/Z39.50 databases).” Actually, Webfeat has translators to thousands of databases that are licensed, proprietary content, the vast majority of which are NOT accessible through SRU/SRW/Z39.50. Only with the announcement of OpenTranslators does that become true.

“1. How “open” are these OpenTranslators?” Our response is that what is “Open” is that we’ve taken open standards and enabled access to the translators. As a result this allows you to leverage existing software applications, programmer’s toolkits, as well as writing your own applications to interact with the translators, through SRU/SRW/Z39.50. Yes, the translators themselves incorporate proprietary technology and will continue to do so. Anyone who has tried to develop a translator will know it is a complicated and demanding process, and that the final product requires constant monitoring and maintenance because back-end interfaces are subject to change at any moment.  Consequently, it is an application that is ideally suited to operate as a hosted service, where revenues is earned from keeping the translators up-to-date and backed bu sufficient CPU cycles to meet requirements.

It is also important to note that we’ve always said that while we believe in OSS, we also know and understand there will be times where we will need to marry OSS and proprietary technologies to provide a total solution. So, bottom line, what we’ve done, is to expose proprietary technology through open, standards-based protocols as a way for end users to pick the metasearch/federated search interface of their choice and still have access to vast amounts of content. It allows developers to focus more attention on interfaces and usability, and less on sorting out how to access a given database. It also dramatically opens the field of search-oriented projects that may be undertaken either by libraries or by startup companies. We believe that this is a dramatic step and one that qualifies as “open”. (We’ll expand on this later in this posting).

“2. How does access to content work when authentication is involved if the connectors to the SRU/SWR/Z39.50 database are WebFeat connectors and there’s a CARE/Index Data gateway in between?” When the service is configured for a customer, WebFeat handles authentication against each database on behalf of the customer. All you need to provide to the OpenTranslator is a customer ID, which is provided to you when you sign on.

“3. What will access cost? What is the licensing model?” As noted elsewhere in the original blog posting, this is a hosted service, so what customers pay is an annual subscription to use both the CARE/ID OpenTranslators and the WebFeat translators. The cost is based on the number of translators used and, as with most such services, the more you buy, the cheaper they get. One yearly cost covers both subscriptions. Interested parties should contact CARE Affiliates for a quote.

“4. Are there really 10,000 databases accessible from this service as advertised in the press release?” WebFeat states that they have 9,000+ databases in their translator library and Index Data provides many thousands more pure Z39.50 connectors. Together, it totals over 10,000. Do some of those databases come from the same content providers? Sure. Like every industry, once you get good at something, you tend to acquire similar companies and the result has been that there is no longer one company per database, nor should there be. Cost efficiencies result from using the same infrastructure to supply access to many different kinds of content. This helps keep costs down for the end users.

“5. Who is hosting access? Is there a plan to handle scaling of the service if it becomes real popular?” Both CARE/Index Data and WebFeat use hosting centers. These kinds of companies offer real benefits for these types of services because they offer redundant processors, telecommunication lines, mirrored disks and instant fail-overs. Plus, they can add additional capacity, usually within hours of a request. So, are we well positioned to scale this service? Absolutely, we’ve planned for it and we’re counting on it.

So, here is a reason why we think OpenTranslators are so important. For OSS to go beyond being an academic practice; beyond being an activity practiced in basements in people’s spare time; and finally beyond being just the sexy word that it is at the moment, then it’s really important that we work out ways to make OSS (and for that matter, open standards, which is really a variation on the theme) coexist with, and indeed support commercial activities. Ultimately that is how things will get done and how usable products will result.

We’re looking for as many areas of synergy as we can, rather than painting OSS as being a threat or even necessarily a total alternative to commercial players. We believe it *should* be viewed as a threat to the stale business practices of many current vendors, but it shouldn’t be viewed as being in opposition to commercial endeavors. OpenTranslators make important headway not only because of what it does for metasearch/federated searching, but also as an example of how total solutions can be assembled using best-of-breed products and companies to deliver what customers want, directly to their desktops. We are crossing new ground that will lead to better products and services for everyone. That is what OpenTranslators do and why they are important.