What can we do right now for a Labs environment?

[Matthew Turk <matthewturk@xxxxxxxxx>]

Hi all,

Glad to see some of the discussion recently (Arthur, I think I owe you a response!) about what we might be able to provide longer term in a "labs" environment that goes beyond a cloud.

I wanted to propose something which may be of interest, which is an extension of the "Epiphyte" demo I showed at the NDS-Rockville meting. A few of us have been working on extending that demo and developing it a bit further, with hardening of some aspects and deploying a few more. But before it got too much further, I wanted to bring it here to see if anyone else was interested in exploring it.

We've been working to set up scripts to automate the process of:

Â* Setting up a CoreOS cluster

Â* Managing docker containers locally

Â* Ingesting data into iRODS

Â* Deploying docker containers dynamically, with access to subsets of iRODS-stored data

Â* Detecting when new containers are exposing port 80, and setting up automatic handling of sub-URLs to those containers

This is an environment in which we would be able to deploy new services for:

Â* Accepting data

Â* Displaying data

Â* Manipulating data

It could be seen as something of a platform-as-a-service for data -- it provides mechanisms for very simply describing services, and then down the road building the matchmaking services that Arthur, Kenton and I were brainstorming over. It could be used to develop frontends to data, to bring data to other frontends, and to re-ingest artifacts for subsequent publication. But it's at its core a set of deployment, management and orchestration mechanisms that are focused around *data* -- and in that way, it should be able to be flexible and experimental.

The use case I've had in my head for a while has been to get some big dataset there, linked to a publication, and then make it easy for people to use IPython notebooks or Python scripts, grab the artifacts those result in, and publish them as data on their own. And I think it's almost there. The next phase, in my eyes, would be to allow people to upload their own webapps that interface with their data. That one might take a bit longer, but not too much.

Anyway, we have a first stab at this running, and next week if you're at SC14 I'll be at the NCSA booth a couple times and can talk about it (it might be listed as "NDS Labs") but I don't think there's any way it can be successful without figuring out if it's on the right track or if it's missing some ideas. And the only way to figure that out is to try to get other people to try it. So if you want to experiment, let's get you onto the system, and let's figure out what's good about it, what's bad about it, and what would make it better.

-Matt