You can check the web ui for any Mesos master and see the version number in the top left corner (at least for any fairly recent Mesos, ~0.18+). You can check the logs for a master or slave, which will always start with the lines "Build: [date] by [user]" and...
Mesos will try to autodetect docker by running docker version. You can specific an absolute path for the docker executable by passing the --docker=/path/to/docker flag to the slave. There are other docker-specific flags for the slave, like --docker_sandbox_directory, --docker_remove_delay, and --docker_stop_timeout. For more details on those, see https://mesos.apache.org/documentation/latest/configuration/ Mesos currently...
Part of the Mesos philosophy, especially for long-running services, is that a failure in one or more Mesos components should not need to stop the user application. If a slave shuts down and the framework has checkpointing enabled, the executor driver will wait for the slave's --recovery_timeout (default 15min) before...
You have to install all mandatory development libraries for this package. In this message it complains on lack of apr-util-devel. I think yum install apr-util-devel should help....
jenkins,jenkins-plugins,mesos,mesosphere
Have a look at the environment variable LIBPROCESS_IP="IP of the jenkins host". See mesos/chronos#193
Full-disclosure: I currently work at Twitter and I'm involved in both Apache Mesos and Aurora. Mesos uses cases can vary based upon a few dimensions: scale (10 servers vs 10s of thousands), available hardware (dedicated/static or in the public cloud/scalable), and workloads (primarily services, batch, or both). Your list is...
distributed,distributed-computing,mesos,mesosphere
When you declineOffer, you can set an optional Filter with refuse_seconds longer than the default 5s. This will mean that after you decline an offer from a node, that Mesos will not offer these resources back to your framework for refuse_seconds. Alternatively, if your framework temporarily doesn't want any offers...
The DCOS Dashboard is pretty cool :-). Currently it is just available via the DCOS beta on AWS and Azure. There will be on prem packages later on as well, potentially even a community edition. Feel free to contact/follow Mesosphere for updates. Until then you can use the standard Mesos,...
docker,mariadb,coreos,mesos,marathon
After posting the question, I was lucky and came across a few repositories that have achieved what I am looking for: Redis https://github.com/mdevilliers/docker-rediscluster - A Redis cluster with two Redis instances and three Redis Sentinel monitors. If the Master fails, the Sentinels promote the Slave as a Master. Mark has...
osx,homebrew,osx-yosemite,mesos
aclocal: error: aclocal: file '/usr/local/share/aclocal/libmcrypt.m4' does not exist That file is provided by MCrypt. brew install mcrypt...
I installed subversion by running brew install subversion --universal and a search of brew packages didn't reveal anything with libsvn in the name. You should be able to see subversion with the brew list command. The issue is that configure is trying to find the headers in /usr/include, but...
mesos,kubernetes,google-container-engine
Kubernetes actually does do this. The master schedules each pod onto a worker node, and each node communicates with the master to receive the work it should do. The scheduler does bin packing based on resource requests. We're working on improving the scheduler to use resource usage as well (what...
mysql,json,apache,docker,mesos
Here's some example Marathon JSON for using Docker's bridged networking mode: { "id": "bridged-webapp", "cmd": "python3 -m http.server 8080", "cpus": 0.5, "mem": 64.0, "instances": 2, "container": { "type": "DOCKER", "docker": { "image": "python:3", "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0, "servicePort": 9000, "protocol": "tcp" }, { "containerPort": 161,...
hadoop,mapreduce,hdfs,mesos,mesosphere
Absolutely possible. Don't think Hadoop as an installable program, it's just composed by a bunch of java processes running on different nodes inside a cluster. If you use hadoop tar ball, you can just run NameNode and DataNodes processes if you only want HDFS. If you use other hadoop distros...
Install mesoshere. You can get the the instructions here.After installation run mesos-master on master and mesos-slave on slaves with ip that point to your master. If you get port already used error as mentioned by Adam above, try running slave on a different port using argument --port=value. If you get...
amazon-web-services,mesos,amazon-vpc,docker-registry,mesosphere
Since the Mesos docs are pretty poor on this, I'm going to answer this wiki-style and update this answer as I go. Strategies that should work Host it on S3 (with networking-based access restrictions) Host the .dockercfg file on S3. For better security, you should consider putting it in its...
tomcat,docker,mesos,mesosphere,marathon
In your JSON file you should have "cmd": "" (last line) otherwise the docker container command is overridden.
This is a great and frequently asked question. Apache Aurora does not currently have a first-class scheduler HTTP/JSON API, but it is being planned (AURORA-987) and an API is one of several ideas that has discussed as a potential projects that may be worked on during a June Aurora community...
apache,compilation,elasticsearch,mesos
Try building once more as per the getting_started guide of mesos. It will create libmesos.so. You may have to provide root privilleges while running make install.
thanks for trying mesos under AIX! There is no official support for AIX,so unfortunately also no documentation what might be potential problems. Did you try building mesos (make && make check) on AIX? That should give a good indicator for potential problems. To your LXC questions: Mesos supports a number...
postgresql,distributed-computing,mesos,mesosphere
Excellent question. Here are a few upcoming features in Mesos to improve support for stateful services, and corresponding current workarounds. Persistent volumes (0.22?): When launching a task, you can create a volume that exists outside of the task's sandbox and will persist on the node even after the task dies/completes....
Depends on what VM networking mode you're using. If the VM has an external IP (like 10.0.2.15), then you may be able to reach Mesos on http://10.0.2.15:5050, otherwise you may need to set up port forwarding or bridged/host-only networking modes. Check out this article on VM (virtualbox) networking for more...
It's unclear what JSON you're referring too, so I'll answer both the case where you want the JSON that describes the application itself (1) (which is used by Marathon) and the case where you're looking to retrieve some JSON output of an application once it has run (2). 1) If...
ubuntu,ubuntu-14.04,mesos,mesosphere
Mesos does not uses a consolidated configuration file. All configuration options you want to set you can set either through environment variables or by creating "option" files in Mesos config directory (/etc/mesos-master/ in my case). For example if you want to change --work_dir option you can do one of the...
distributed-computing,distributed-system,mesos,mesosphere
The default timeout for an executor with an slave is 1 minute and can be changed with the --executor_registration_timeout slave flag. From Mesos Configuration --executor_registration_timeout=VALUE Amount of time to wait for an executor to register with the slave before considering it hung and shutting it down (e.g., 60secs, 3mins, etc)...
There are a number of ways to get your application to run on DCOS (and/or Mesos). First for legacy applications you can use the marathon framework which you can view as kind of the init system of DCOS/Mesos. If you need more elaborate applications and want to really program against...
The Logis not about accessing log messages, but about storing data in Mesos own replicated log (a distributed data storage). see also: http://mesos.apache.org/blog/mesos-0-17-0-released-featuring-autorecovery/...
twitter,cluster-computing,gpgpu,mesos,mesosphere
Mesos does not yet provide direct support for (GP)GPUs, but does support custom resource types. If you specify --resources="gpu(*):8" when starting the mesos-slave, then this will become part of the resource offer to frameworks, which can launch tasks that claim to use these resources. Once some of the gpu resources...
All logs are stored in the log directory you specify via a --log_dir flag. I'm not sure you can access logs from previous runs via WebUI, but you can definitely ssh into the specific machine.
amazon-ec2,ubuntu-14.04,mesos,mesosphere
Mesos uses gethostbyname2 to resolve hostnames to IPs. The first thing I would recommend, is to try "ping localhost" and "ping hostname", and verify that there are no strange settings in /etc/hosts. If you're doing a multi-node cluster, I'd recommend that hostname map to the public IP address (not 127.0.x.1)....
Even after specifying the ports resource Disk, CPU, Mem should be added automatically (see log excerpt) below. mesos-slave.sh --master=xxxx:5050 --resources=ports:80 I0503 20:48:04.999114 2057073408 main.cpp:200] Starting Mesos slave I0503 20:48:05.000370 243535872 slave.cpp:316] Slave resources: ports(*):80; cpus(*):8; mem(*):15360; disk(*):470848 Unfortunately the ports are not additive anymore... Maybe one idea could be to...
java,caching,apache-spark,mesos
Vainilla Spark is not able to do this (at the time of writing - Spark is evolving fast) . There's a Spark-JobServer contributed by Ooyala that exactly fulfills your needs. It keeps a register with the jars for sequential job submission and provides additional facilities to cache RDDs by name....
I think the best to answer that are those who work on Spark. So, from Learning Spark Start with a standalone cluster if this is a new deployment. Standalone mode is the easiest to set up and will provide almost all the same features as the other cluster managers if...
elasticsearch,docker,mesos,marathon
Elasticsearch and NFS are not the best of pals ;-). You don't want to run your cluster on NFS, it's much too slow and Elasticsearch works better when the speed of the storage is better. If you introduce the network in this equation you'll get into trouble. I have no...
hadoop,docker,cluster-computing,mesos,marathon
You should be able to run it, taking care of some issues when running the mesos (with Docker) containers, like running in privileged mode. Take a look to jpetazzo/dind to see how you can install and run docker in docker. Then you can setup mesos in that container to have...
Short anwer: No. Unfortunately, Spark Mesos and YARN only allow giving as much resources (cores, memory, etc.) per machine as your worst machine has (discussion). Ideally, the cluster should be homogeneous in order to take full advantage of its resources. However, there might exist a workaround for your problem. According...
It is safe to say that Mesos assumes that all your final data is stored somewhere when you task finishes, and it's your, or if you want, your task's or your framework's responsibility to ensure this. If you want to persist intermediate results, or share results between tasks, you can...
Checked executor logs (stderr in /tmp/mesos/slaves/...) and found out that JAVA_HOME was not set, so the hadoop dfs command was not able to run to fetch the executor. The URI was perfect, the JAVA_HOME was not set. Additionally I had to set HADOOP_HOME when starting the slaves.
Yes, it's possible, see Application Deployments. I'm in the process of extending the Marathon doc to make this even clearer. Will provide updates here as well.
dns,docker,mesos,marathon,consul
I have never tried to use the $HOST variable in the parameters list but you can use it in the cmd specifier. You can do something like this: "cmd": "python3 myscript.py $HOST", You will get the hostname or the ip address of your mesos slave as the first program argument...
apache,elasticsearch,zookeeper,mesos,mesosphere
According to the README at https://github.com/mesosphere/elasticsearch-mesos, you may need to modify mesos.master.url to point to the same ZK url that the Mesos master is using (maybe not localhost). If you're using a single-master Mesos cluster, you can skip the ZK url and point this parameter directly to the Mesos master....
ubuntu,gnupg,apt-get,mesos,openpgp
apt uses it's own GnuPG keyring /etc/apt/trusted.gpg. Use apt-key to fetch the key instead, which wraps around GnuPG using the right keyring. sudo apt-key adv --keyserver keyserver.ubuntu.com --recv DF7D54CBE56151BF This is also what's proposed in the official documents, but I replaced the short key ID with a long key ID...
Mesos only supports "Posix" isolation on Macs, which is really more for monitoring than actual resource limiting, so testing Mesos on a Mac will not be able to demonstrate true resource isolation. If you test Mesos on a Linux machine/VM, you can set --isolation='cgroups/cpu,cgroups/mem' when starting each slave to enable...
Mesos itself does a lot of things unrelated to Docker: While there is support for containers (see http://mesos.apache.org/documentation/latest/docker-containerizer/), there is no direct dependency from Mesos to Docker. When you look at the architecture example (from http://mesos.apache.org/documentation/latest/mesos-architecture/): you can see that neither the master nor any of the slaves do anything...
Great questions! I believe it would be really helpful to have a look at a sample framework such as Rendler. This will probably answer most of your question and give you feeling for the framework internal. Let me now try to answer the question which might be still be open...
If you want to link two containers, they have to run on the same host. Marathon currently does not support that. To me it sounds like you don't really need that. Instead you could just use service discovery mechanisms to make your db accessible to the other service. For this...
Try sending SIGUSR1 to the slave process. As an alternative, you can also simply kill the process and remove slave's working directory, the Mesos Master will forget about this slave and all executors will die after some period of time. However, it's not that "clean" in comparison to the previous...
docker,mesos,mesosphere,marathon
You can add constraints to your app to force them onto specific hosts. In a simple case, you could do something like this to force an app onto a specific host: { "instances": 1, "constraints": [["hostname", "LIKE", "worker-1"]] } The next option brings in attributes. You can tag a known...
java,cluster-computing,virtualization,mesos,iaas
Your summary is almost right but it does not reflect the essence of what mesos represents. The vision of mesosphere, the Company behind the project, is to create a "Datacenter Operating System" and the mesos is the kernel of it in analogy to the kernel of a normal OS. The...