Beyond another cloud: data service discovery for NDSLabs

[Matthew Turk <matthewturk@xxxxxxxxx>]

Hi all,

In the other thread, Arthur brought up that we don't want "just another cloud infrastructure," which I think was really apt, and something that deserves thought for any NDS Labs project. So I wanted to start a couple topics about what can be provided on top of a standard cloud infrastructure that might be of use.

I'm wondering about discovering data services within a region, where that region is either some subnet on a cloud provider, or even more globally across locations. If we are thinking about interoperability of services, then there are probably a few verbs that could be identified as being necessary. If we can have services identify themselves as providing verb endpoints, that could provide an environment for testing interop.

Kacper and I have been experimenting with this ourselves, mostly looking at the various service discovery mechanisms that operate on docker containers being orchestrated across machines. Some of these do this via introspection, and some will even set up automatic (nginx) reverse proxies for docker containers running inside a system. Right now it looks like etcd is a pretty good solution for this:

https://coreos.com/docs/distributed-configuration/getting-started-with-etcd/

http://www.activestate.com/blog/2014/03/brandon-philips-explains-etcd

https://github.com/coreos/etcd

as it can allow for key/value pairs to be stored, and it's discoverable. For instance:

http://jasonwilder.com/blog/2014/07/15/docker-service-discovery/

I think having a discussion about what we want services to be able to do is perhaps a much bigger topic, but I wonder if this type of thing -- particularly etcd -- would be useful to any projects, and would be a good avenue for service discovery and intertop. Is there something else that would be better?

-Matt