Docker on Mesos: Volume is placed on which node?
It is safe to say that Mesos assumes that all your final data is stored somewhere when you task finishes, and it's your, or if you want, your task's or your framework's responsibility to ensure this. If you want to persist intermediate results, or share results between tasks, you can...
What does Google Container Engine use for Kubernetes node provisioning?
mesos,kubernetes,google-container-engine
Kubernetes actually does do this. The master schedules each pod onto a worker node, and each node communicates with the master to receive the work it should do. The scheduler does bin packing based on resource requests. We're working on improving the scheduler to use resource usage as well (what...
Mesos webUI display only the recently connected slave
Install mesoshere. You can get the the instructions here.After installation run mesos-master on master and mesos-slave on slaves with ip that point to your master. If you get port already used error as mentioned by Adam above, try running slave on a different port using argument --port=value. If you get...
How to use the org.apache.mesos.Log?
The Logis not about accessing log messages, but about storing data in Mesos own replicated log (a distributed data storage). see also: http://mesos.apache.org/blog/mesos-0-17-0-released-featuring-autorecovery/...
It's possible only install Hadoop HDFS?
hadoop,mapreduce,hdfs,mesos,mesosphere
Absolutely possible. Don't think Hadoop as an installable program, it's just composed by a bunch of java processes running on different nodes inside a cluster. If you use hadoop tar ball, you can just run NameNode and DataNodes processes if you only want HDFS. If you use other hadoop distros...
How to fetch OpenPGP key for Mesos installation?
ubuntu,gnupg,apt-get,mesos,openpgp
apt uses it's own GnuPG keyring /etc/apt/trusted.gpg. Use apt-key to fetch the key instead, which wraps around GnuPG using the right keyring. sudo apt-key adv --keyserver keyserver.ubuntu.com --recv DF7D54CBE56151BF This is also what's proposed in the official documents, but I replaced the short key ID with a long key ID...
Apache Mesos's Docker Containerizer
Mesos will try to autodetect docker by running docker version. You can specific an absolute path for the docker executable by passing the --docker=/path/to/docker flag to the slave. There are other docker-specific flags for the slave, like --docker_sandbox_directory, --docker_remove_delay, and --docker_stop_timeout. For more details on those, see https://mesos.apache.org/documentation/latest/configuration/ Mesos currently...
Linking containers with Mesos JSON file
If you want to link two containers, they have to run on the same host. Marathon currently does not support that. To me it sounds like you don't really need that. Instead you could just use service discovery mechanisms to make your db accessible to the other service. For this...
there is an error about libtool when building mesos
You have to install all mandatory development libraries for this package. In this message it complains on lack of apr-util-devel. I think yum install apr-util-devel should help....
Is a LXC environment a mandatory requirement for operating mesos/sphere
thanks for trying mesos under AIX! There is no official support for AIX,so unfortunately also no documentation what might be potential problems. Did you try building mesos (make && make check) on AIX? That should give a good indicator for potential problems. To your LXC questions: Mesos supports a number...
Is it possible to cache the application jar when running Apache Spark applications on a cluster?
java,caching,apache-spark,mesos
Vainilla Spark is not able to do this (at the time of writing - Spark is evolving fast) . There's a Spark-JobServer contributed by Ooyala that exactly fulfills your needs. It keeps a register with the jars for sequential job submission and provides additional facilities to cache RDDs by name....
Mesos master setup, but home page does not return
Depends on what VM networking mode you're using. If the VM has an external IP (like 10.0.2.15), then you may be able to reach Mesos on http://10.0.2.15:5050, otherwise you may need to set up port forwarding or bridged/host-only networking modes. Check out this article on VM (virtualbox) networking for more...
How to stop mesos from offering resources to a framework?
distributed,distributed-computing,mesos,mesosphere
When you declineOffer, you can set an optional Filter with refuse_seconds longer than the default 5s. This will mean that after you decline an offer from a node, that Mesos will not offer these resources back to your framework for refuse_seconds. Alternatively, if your framework temporarily doesn't want any offers...
Apache Mesos Schedulers and Executors by example
Great questions! I believe it would be really helpful to have a look at a sample framework such as Rendler. This will probably answer most of your question and give you feeling for the framework internal. Let me now try to answer the question which might be still be open...
How should a .dockercfg file be hosted in a Mesosphere-on-AWS setup so that only Mesosphere can use it?
amazon-web-services,mesos,amazon-vpc,docker-registry,mesosphere
Since the Mesos docs are pretty poor on this, I'm going to answer this wiki-style and update this answer as I go. Strategies that should work Host it on S3 (with networking-based access restrictions) Host the .dockercfg file on S3. For better security, you should consider putting it in its...
jenkins slave in mesos not start
jenkins,jenkins-plugins,mesos,mesosphere
Have a look at the environment variable LIBPROCESS_IP="IP of the jenkins host". See mesos/chronos#193
Docker container running Mesos cluster and running other docker containers on cluster (using Marathon)
hadoop,docker,cluster-computing,mesos,marathon
You should be able to run it, taking care of some issues when running the mesos (with Docker) containers, like running in privileged mode. Take a look to jpetazzo/dind to see how you can install and run docker in docker. Then you can setup mesos in that container to have...
Creating a new app Marathon App from web browser
It's unclear what JSON you're referring too, so I'll answer both the case where you want the JSON that describes the application itself (1) (which is used by Marathon) and the case where you're looking to retrieve some JSON output of an application once it has run (2). 1) If...
How to see the history log Message of Mesos when Master or Slave restart
All logs are stored in the log directory you specify via a --log_dir flag. I'm not sure you can access logs from previous runs via WebUI, but you can definitely ssh into the specific machine.
Deploying Docker containers with port-mapping on Mesos/Marathon
mysql,json,apache,docker,mesos
Here's some example Marathon JSON for using Docker's bridged networking mode: { "id": "bridged-webapp", "cmd": "python3 -m http.server 8080", "cpus": 0.5, "mem": 64.0, "instances": 2, "container": { "type": "DOCKER", "docker": { "image": "python:3", "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0, "servicePort": 9000, "protocol": "tcp" }, { "containerPort": 161,...
How to see the installed version of Apache Mesos?
You can check the web ui for any Mesos master and see the version number in the top left corner (at least for any fairly recent Mesos, ~0.18+). You can check the logs for a master or slave, which will always start with the lines "Build: [date] by [user]" and...
docker mesosphere marathon - unable to see running process inside a docker container
tomcat,docker,mesos,mesosphere,marathon
In your JSON file you should have "cmd": "" (last line) otherwise the docker container command is overridden.
What does Apache Mesos actually do?
java,cluster-computing,virtualization,mesos,iaas
Your summary is almost right but it does not reflect the essence of what mesos represents. The vision of mesosphere, the Company behind the project, is to create a "Datacenter Operating System" and the mesos is the kernel of it in analogy to the kernel of a normal OS. The...
Does Apache Mesos recognize GPU cores?
twitter,cluster-computing,gpgpu,mesos,mesosphere
Mesos does not yet provide direct support for (GP)GPUs, but does support custom resource types. If you specify --resources="gpu(*):8" when starting the mesos-slave, then this will become part of the resource offer to frameworks, which can launch tasks that claim to use these resources. Once some of the gpu resources...
Is there a timeout for an executor to register with Mesos master?
distributed-computing,distributed-system,mesos,mesosphere
The default timeout for an executor with an slave is 1 minute and can be changed with the --executor_registration_timeout slave flag. From Mesos Configuration --executor_registration_timeout=VALUE Amount of time to wait for an executor to register with the slave before considering it hung and shutting it down (e.g., 60secs, 3mins, etc)...
Hadoop 2.5.2 on Mesos 0.21.0 - Failed to fetch URIs for container
Checked executor logs (stderr in /tmp/mesos/slaves/...) and found out that JAVA_HOME was not set, so the hadoop dfs command was not able to run to fetch the executor. The URI was perfect, the JAVA_HOME was not set. Additionally I had to set HADOOP_HOME when starting the slaves.
Why marathon does not terminate jobs after the quorum is lost?
Part of the Mesos philosophy, especially for long-running services, is that a failure in one or more Mesos components should not need to stop the user application. If a slave shuts down and the framework has checkpointing enabled, the executor driver will wait for the slave's --recovery_timeout (default 15min) before...
Mesos: mesos-slave offer additional ports
Even after specifying the ports resource Disk, CPU, Mem should be added automatically (see log excerpt) below. mesos-slave.sh --master=xxxx:5050 --resources=ports:80 I0503 20:48:04.999114 2057073408 main.cpp:200] Starting Mesos slave I0503 20:48:05.000370 243535872 slave.cpp:316] Slave resources: ports(*):80; cpus(*):8; mem(*):15360; disk(*):470848 Unfortunately the ports are not additive anymore... Maybe one idea could be to...
Mesos real world use-cases
Full-disclosure: I currently work at Twitter and I'm involved in both Apache Mesos and Aurora. Mesos uses cases can vary based upon a few dimensions: scale (10 servers vs 10s of thousands), available hardware (dedicated/static or in the public cloud/scalable), and workloads (primarily services, batch, or both). Your list is...
does Mesos provide its service like cluster management UI as OSS project?
The DCOS Dashboard is pretty cool :-). Currently it is just available via the DCOS beta on AWS and Azure. There will be on prem packages later on as well, potentially even a community edition. Feel free to contact/follow Mesosphere for updates. Until then you can use the standard Mesos,...
How can I compile libmesos.so file?
apache,compilation,elasticsearch,mesos
Try building once more as per the getting_started guide of mesos. It will create libmesos.so. You may have to provide root privilleges while running make install.
Does Apache Aurora have API?
This is a great and frequently asked question. Apache Aurora does not currently have a first-class scheduler HTTP/JSON API, but it is being planned (AURORA-987) and an API is one of several ideas that has discussed as a potential projects that may be worked on during a June Aurora community...
How to port existing distributed service application to DCOS?
There are a number of ways to get your application to run on DCOS (and/or Mesos). First for legacy applications you can use the marathon framework which you can view as kind of the init system of DCOS/Mesos. If you need more elaborate applications and want to really program against...
How to configure a high-availability cluster of MariaDB and Redis in Mesos or CoreOS
docker,mariadb,coreos,mesos,marathon
After posting the question, I was lucky and came across a few repositories that have achieved what I am looking for: Redis https://github.com/mdevilliers/docker-rediscluster - A Redis cluster with two Redis instances and three Redis Sentinel monitors. If the Master fails, the Sentinels promote the Slave as a Master. Mark has...
Unable to get Mesos to run from tutorial: Setting up a Single Node Mesosphere Cluster
amazon-ec2,ubuntu-14.04,mesos,mesosphere
Mesos uses gethostbyname2 to resolve hostnames to IPs. The first thing I would recommend, is to try "ping localhost" and "ping hostname", and verify that there are no strange settings in /etc/hosts. If you're doing a multi-node cluster, I'd recommend that hostname map to the public IP address (not 127.0.x.1)....
How does Marathon limit resources for directly called apps?
Mesos only supports "Posix" isolation on Macs, which is really more for monitoring than actual resource limiting, so testing Mesos on a Mac will not be able to demonstrate true resource isolation. If you test Mesos on a Linux machine/VM, you can set --isolation='cgroups/cpu,cgroups/mem' when starting each slave to enable...
Which cluster type should I choose for Spark?
I think the best to answer that are those who work on Spark. So, from Learning Spark Start with a standalone cluster if this is a new deployment. Standalone mode is the easiest to set up and will provide almost all the same features as the other cluster managers if...
How do you compile Mesos on OS X Yosemite
osx,homebrew,osx-yosemite,mesos
aclocal: error: aclocal: file '/usr/local/share/aclocal/libmcrypt.m4' does not exist That file is provided by MCrypt. brew install mcrypt...
Persistent storage for Apache Mesos
postgresql,distributed-computing,mesos,mesosphere
Excellent question. Here are a few upcoming features in Mesos to improve support for stateful services, and corresponding current workarounds. Persistent volumes (0.22?): When launching a task, you can create a volume that exists outside of the task's sandbox and will persist on the node even after the task dies/completes....
Strategy to persist the node's data for dynamic Elasticsearch clusters
elasticsearch,docker,mesos,marathon
Elasticsearch and NFS are not the best of pals ;-). You don't want to run your cluster on NFS, it's much too slow and Elasticsearch works better when the speed of the storage is better. If you introduce the network in this equation you'll get into trouble. I have no...
How to use variable in Marathon Docker parameters
dns,docker,mesos,marathon,consul
I have never tried to use the $HOST variable in the parameters list but you can use it in the cmd specifier. You can do something like this: "cmd": "python3 myscript.py $HOST", You will get the hostname or the ip address of your mesos slave as the first program argument...
How do I cleanly shut down a mesos-slave?
Try sending SIGUSR1 to the slave process. As an alternative, you can also simply kill the process and remove slave's working directory, the Mesos Master will forget about this slave and all executors will die after some period of time. However, it's not that "clean" in comparison to the previous...
how know container name with marathon rest API
docker,mesos,mesosphere,marathon
You can add constraints to your app to force them onto specific hosts. In a simple case, you could do something like this to force an app onto a specific host: { "instances": 1, "constraints": [["hostname", "LIKE", "worker-1"]] } The next option brings in attributes. You can tag a known...
Mesos master configuration error: Failed to load unknown flag
ubuntu,ubuntu-14.04,mesos,mesosphere
Mesos does not uses a consolidated configuration file. All configuration options you want to set you can set either through environment variables or by creating "option" files in Mesos config directory (/etc/mesos-master/ in my case). For example if you want to change --work_dir option you can do one of the...
Is Docker native on Mesos OS X?
Mesos itself does a lot of things unrelated to Docker: While there is support for containers (see http://mesos.apache.org/documentation/latest/docker-containerizer/), there is no direct dependency from Mesos to Docker. When you look at the architecture example (from http://mesos.apache.org/documentation/latest/mesos-architecture/): you can see that neither the master nor any of the slaves do anything...
How to install libsvn-dev for mesos from git on OS X mavericks
I installed subversion by running brew install subversion --universal and a search of brew packages didn't reveal anything with libsvn in the name. You should be able to see subversion with the brew list command. The issue is that configure is trying to find the headers in /usr/include, but...
Fig / Docker-Compose-like JSON to feed into Mesosphere/Marathon to setup Multi-Tier Application
Yes, it's possible, see Application Deployments. I'm in the process of extending the Marathon doc to make this even clearer. Will provide updates here as well.
elasticsearch-mesos not getting listed under frameworks of mesosUI
apache,elasticsearch,zookeeper,mesos,mesosphere
According to the README at https://github.com/mesosphere/elasticsearch-mesos, you may need to modify mesos.master.url to point to the same ZK url that the Mesos master is using (maybe not localhost). If you're using a single-master Mesos cluster, you can skip the ZK url and point this parameter directly to the Mesos master....
Spark executors with different amounts of memory on Mesos
Short anwer: No. Unfortunately, Spark Mesos and YARN only allow giving as much resources (cores, memory, etc.) per machine as your worst machine has (discussion). Ideally, the cluster should be homogeneous in order to take full advantage of its resources. However, there might exist a workaround for your problem. According...