The LINGUIST List Operation

Dear LINGUIST List supporters,

Many of you have heard that the LINGUIST List relocated from Eastern Michigan University in Ypsilanti to Indiana University in Bloomington in 2014. Please allow us to summarize what this relocation involved.

In spring 2014 we started cleaning out the former space of LINGUIST List and the Institute for Language Information and Technology (ILIT) at EMU and planning the relocation to Indiana University. Some team members decided to join us in the relocation and continue their work and lives at the new location. Unfortunately, not everybody could join us. Our editors Uliana and Danuta continue to support LINGUIST List remotely, but decided to stay in Michigan.

As you can imagine, the LINGUIST List operation involves a significant amount of technology and equipment. The servers that the LINGUIST List was using in Michigan supported among others the following systems:

• The web server for various web pages and sites (based on numerous instances of the Apache web/httpd server)
• A LISTSERV list server using the commercial software system sold by L-Soft serving not only LINGUIST and LINGLITE, but also numerous other linguistic mailing lists
• An Oracle database server that contained all the website information and project data, from the LINGUIST List issues and postings, to dictionaries and linguistic maps and polygons for the GIS system
• An Adobe ColdFusion web application server for the main website and some project pages
• Apache Tomcat servers for Java-applications
• A GIS server using the GeoServer system to serve the unique collection of historic linguistic maps, the LL-Map project
• Various Ruby on Rails and PHP applications using MySQL, like: ODIN, LEGO, GOLD, MultiTree
• An email server
• A file server for large file collections, project data, internal documents, collaborative project material
• Etc.

It was clear that it would not be possible to relocate the hardware (7 servers of varying age and capacity) and the other equipment. One of the problems we were facing was that policies and restrictions at our new hosting institution would not allow us to operate the respective servers there. It also quickly became clear that the LINGUIST List would not have the funds to pay the expensive licenses for commercial software, e.g. LISTSERV, Adobe ColdFusion, or Oracle database server at the new location.

Many of our online linguistic tools (e.g. LEGO, GOLD, etc.) were developed long ago, with funds from research grants, using now outdated software, with systems running untouched for years on outdated infrastructure, written in programming languages that have been overhauled ever since and so on. As in any research environment with IT-systems and software, as soon as the software is ready and installed, the environment, programming language, and systems are outdated and need updates. For many systems, we were facing the situation that they could not be updated at all anymore, since they relied on components that were removed from modern Linux distributions years ago, because the programming languages and libraries they used were not even available anymore (in the required version).

All these issues together posed a serious problem. LINGUIST had no resources to fund new servers or the redevelopment and adaptation of the software and applications. No research funding agency could be approached in such a short time to help find a solution and preserve the data and applications. LINGUIST had no funds for a basic IT-infrastructure, or the mentioned commercial software licenses for the existing infrastructure and organization. On the other hand, the basics to run an operation like LINGUIST and all the projects and online applications were missing. The infrastructure demands are huge, e.g. a large digital storage space and quite high computational power to cope with the amount of data are needed, to serve millions of access requests every day, handle large amounts of data transfer, etc. On the other hand, the labor necessary to handle the setup, installation, administration, programming and data management was just overwhelming and immense. We had no funds to support any external IT-person to help us with the launch of the systems and services.

As you can imagine, in addition to these problems, there was no available solution to get help with these technical problems. There was not even time to ask for help, to start a new fund drive, or explain to willing helpers and volunteers what needs to be done, and how one could help us. As we were running out of funds, we were running out of time. We were already in over our heads.

Just before the move there were two significant steps that we took. We asked companies for help. We approached Google with an application to grant us free access to their applications and services as a Charitable Non-Profit organization. They approved us. Our problems with data storage, operational email and management tools were solved. We approached GitHub and Bitbucket to grant us free access to their services to manage our code-base for all the systems and software development projects that we had, and quite many we had… Since Bitbucket approved our application first, we decided to go with their service. We are grateful that Google and Bitbucket decided to support us and significantly reduce the workload that we had. Software development with the help of services like GitHub or Bitbucket is significantly easier and faster. We have a very good versioning system now, and collaboration between team members and external helpers is much, much better.

Since various policies at the new institution do not allow us to operate our own list- or email-server within the hosting institution’s intranet, we had to set up the necessary servers outside of the institution through commercial means. We also had to find fast and easy solutions for the LINGUIST List website and various other services to minimize the downtime during the move as much as possible. We have chosen to use Amazon EC2 and A2 Hosting virtual servers for that. These virtual server instances have significant advantages, but they also come with a price-tag. The price for the virtual servers is still lower than investing in new hardware, server hosting at any location, and hardware maintenance and administration costs. We estimate the LINGUIST List saves significantly on operational costs with the new infrastructure. In addition to that, the virtual server infrastructure opens up new flexible solutions. Any server instance can be backed up as an image, that we can download and even run in a virtualization software tool on our desktop machines. The new management tools for tablets for example offer an easy and neat administration interface. It has the touch of Star Trek to open up the tablet and add a new CPU or more memory to the servers, reboot the machine from a mobile phone, and so on.

The LINGUIST List team decided to stick with Linux as the operating system for all servers. We also decided to use only open source and free software for everything from now on. The database was replaced by PostgreSQL. The LISTSERV system was replaced by Mailman. Adobe ColdFusion was replaced with the open source and free Railo system. All operating systems were replaced by free and open Debian-based Linux systems. Even the desktop systems for the editors, developers and managers were replaced by Linux PCs. Our development environment is based on Vim, Eclipse, and other open and free tools. We have to confess, we make use of PyCharm (the free and community, or student and faculty edition, thanks to JetBRAINS for providing those free of charge).

The changes from a commercial database software to an open source one, or the switch from Adobe ColdFusion to Railo, do not just mean no licensing fees and therefore savings. They actually came with an incredible investment upfront. Most of the code, all SQL database commands and code sequences, the ColdFusion code – essentially everything had to be checked and rewritten. This could not be done in a month, two months or half a year. Given the aforementioned problems with hardware, outdated software, and other finance and time problems, this was just a very bad move. We cannot switch at the same time the running systems to free and open ones. Well, we can, and we did. Since we had to invest in updating the systems anyway, we thought that we can also rewrite and change everything and make the move to Open and Free. We have rewritten so much of the old vintage LINGUIST List website, it is an entirely new system in the back-end. We paid for the switch from commercial and expensive software to free and open source systems with our free time. We invested our weekends, nights, and holidays in the port and the relocation. More than once we had reached a point of total frustration, of physical and mental exhaustion, where no more coffee or sugar resources would help. Can you imagine? At the same time, we had to run the operations, continue editing, posting, talk to colleagues who want to make changes of postings, job ads, conference announcements, and also rent trucks for the relocation, commute back and forth for negotiations, checking out new housing and office spaces etc. May to August 2014 were the wildest months of our lives.

Many of you have experienced some glitches and broken or dysfunctional pages. We are sorry for that. Given the short time for relocation and the switch of the paradigms and systems, we were not able to test upfront before bringing up and making it live, but rather had to use user feedback to fix issues as they occurred. We transferred the lists to the new Mailman system. This has caused some of the deactivated accounts to be activated again. Colleagues and subscribers started getting mails and were quite surprised to receive the full LINGUIST list email collection every day; some were even angry with us. We are sorry for causing you this inconvenience, but there was no other way for us to transfer the list server mails, archives and subscriptions to the new system.

The team at LINGUIST List was massively reduced. Only Malgosia, Lwin, and Damir relocated from Michigan to Indiana, together with three GAs, Andrew, Sara, and Anna. The relocation meant not only a relocation of families, children, and households, it also meant the relocation of resources, the acquisition of equipment, the setting up of a new office space for the operation, and also the cleaning up the old one. The team did an incredible job. Within just 6 months all that was accomplished, and the operation of LINGUIST List was interrupted just for some hours and minutes. Many people did not realize that. Many in fact feared that this endeavor will fail, that it was basically impossible to achieve all this in such a short time.

We are lucky that IU provided us with a nice building to restart our operations. We were able to acquire a few PCs to start working again and we got some furniture from surplus to equip a meeting room and basic office space. We have a coffee machine again in the office, and things have calmed down somewhat. We sleep again, and life has some rhythm again. There is still a lot of work, a lot to do, and a lot we need to arrange and organize.

In the meantime we can report that:

• The LINGUIST List website is up and running, faster and more stable than before, not only the newly written ‘vintage website’ with the new PostgreSQL database and Railo ColdFusion engine, but also the new website, which we could not continue developing since spring 2014 (based on Django and Python) because of the move.
• EMELD is up and running, with some minor issues to fix from time to time. The code has been transfered from Adobe ColdFusion and Oracle to Railo and PostgreSQL.
• The list server is back and all the archives and other functionalities are up, hosting not just LINGUIST and LINGLITE, but also many other lists that some of you might be subscribed to. We are now using the GNU open source server Mailman.
• LEGO is up, with some issues that we still need to fix. This site was written in PHP and specific extensions and libraries. It uses in the backend the Apache Solr indexing engine running on a Tomcat server. This was a lot of work, to reinstall it and set it up. Some minor issues need to be fixed that have to do with the Solr communication in searches.
• GOLD is up and running. It was also written in PHP using the Zend framework. We had to port old code to new server and software environments.
• MultiTree is up, both the new and the old site. The old system still needs to be fixed, and the new one that was developed using Django and D3js needs some more development. The old system was written in one of the early Ruby on Rails versions. The port to the more recent Railo versions was quite complex.
• OLAC is connected again, thanks to the help of many colleagues, e.g. Gary Simons, Steven Bird and others.
• ODIN is up, and needs some minor corrections.
• LL-Map is installed and needs to be activated again. Soon we should have the system and the connections up again, and all the polygons and maps available for browsing and search, linked to MultiTree, even LEGO and GOLD etc. There are new ways to contribute own maps and information now.
• Etc.

There is still a lot to do. Most of the transfer has been accomplished. We did everything we could to preserve the data, port the applications, make the new site and operations more sustainable, cheaper, more open, and robust.

We are all set for a new start. After 25 years of the LINGUIST List, the technology and environment is again up to date, ready for the next 25 years.

Many of you know, the LINGUIST List has a very low operational budget. It did operate at its financial limits since spring 2013, without a fund drive in 2013, and a limited fund drive in 2014. LINGUIST started in the new location without any significant funds, just with the help and support of its hosting institution, the team, and some supporters.

The team and the operation now need your help. We depend on the Fund Drive 2015 to be able to continue with normal operations during the summer 2015, and during the next academic year. Graduate assistantships do not cover the summer. Although IU supports us with two fully covered GAs, and two partially covered ones (in addition to all the other support that we get from IU and the Department of Linguistics), we need to cover the summer months by paying editors. We also need more person-power to cover the next academic year.

Please consider helping LINGUIST List to continue its operations and donate during the 2015 Fund Drive.

The LINGUIST List Team