Menu
  • HOME
  • TAGS

How Jenkins is passing username and password credentials for checkout pull operations

Tag: jenkins,mercurial,mercurial-extension,maven-scm-plugin,mercurial-keyring

I have Jenkins / Mercurial latest versions. Machine is Red Hat Linux 6.6.

I'm using Release plugin in Jenkins. Maven plugins (maven-scm-plugin, maven-version-plugin and maven-enforcer-plugin) for doing release process on a project. All of these versions are using latest available versions and configuration is setup correctly.

In Jenkins job, I'm checking out the source code from a Project which sits behind RhodeCode (Mercurial hg). enter image description here

Output shows like:

06:00:02 Started by timer
06:00:02 [EnvInject] - Loading node environment variables.
06:00:03 Building on master in workspace /main/jenkins/instance2/workspace/MyCoolProject
06:00:06 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" showconfig paths.default
06:00:06 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" pull --rev default
06:00:12 pulling from http://mercurialserver.my.company.com:9001/csa/MyCoolProject/
06:00:12 no changes found
06:00:12 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" update --clean --rev default
06:00:13 0 files updated, 0 files merged, 0 files removed, 0 files unresolved
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" --config extensions.purge= clean --all
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" log --rev . --template {node}
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" log --rev . --template {rev}
06:00:13 [MyCoolProject] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" log --rev 

As you see above, because we specified a user to perform the read/checkout operation in Mercurial DVCS tool, Jenkins passed bunch of parameters while initiating the hg commands i.e.

--config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https"

I'm trying to find out how I can send the same, when I'm using Maven SCM plugin and it's goals.

scm:checkin
scm:tag
etc which when called, initiates the underlaying version control commands (in my case, hg branch, hg outgoing, hg push commands).

Right now, the above hg commands are failing due to the fact that either the user doesn't have ~/.hgrc file containing username and password variable set OR the user does NOT have "WRITE" access to the target RhodeCode/Mercurial repository.

20:16:26 [INFO] --- maven-scm-plugin:1.9.4:checkin (default-cli) @ MyCoolProject ---
20:16:27 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg branch
20:16:27 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg outgoing
20:16:29 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg status
20:16:29 [INFO] [pom.xml:modified]
20:16:29 [INFO] [jenkins-MyCoolProject-43.appVersion.txt:unknown]
20:16:29 [INFO] [pom.xml.versionsBackup:unknown]
20:16:29 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg commit --message '"CM Jenkins - Release plugin auto check-in and creation of release tag = 0.0.29'
20:16:30 [INFO] EXECUTING: /bin/sh -c cd /main/jenkins/instance2/workspace/MyCoolProject && hg push http://cmprod2merc.my.company.com:9001/csa/MyCoolProject
20:16:34 [ERROR] 
20:16:34 EXECUTION FAILED
20:16:34   Execution of cmd : push failed with exit code: 255.
20:16:34   Working directory was: 
20:16:34     /main/jenkins/instance2/workspace/MyCoolProject
20:16:34   Your Hg installation seems to be valid and complete.
20:16:34     Hg version: 1.9.2 (OK)
  • I tried configuring ~/.hgrc file (as per Mercurial docs) and everything works if the username you specify in this file has valid WRITE access on the target source code repository/project then the above errors message won't come.

  • This can also be resolved if I install/configure keyring and mercurial_keyring extension/plugin on the build machine (which is easy as per the online mercurial keyring documentation).

What I'm trying to find out is:
How can I pass the --config xxxxx parameters (like Jenkins is passing while doing checkout / pull operations - coming due to the setting values we have entered under Source Code Management in Jenkins job) to the hg commands which are called when Maven SCM plugin is initiating scm:checkin / scm:tag goals which ends up calling hg commands (hg outgoing, hg push) / all hg commands without setting ~/.hgrc and installing mercurial_keyring?

Best How To :

Found the solution:

  1. Even when the user (jenkins) had valid WRITE access on the target RhodeCode/Merurial repository and either ~/.hgrc or mercurial_keyring setup wsa configured successfully (i.e. username/password less hg operations after one time manual entry), Maven SCM plugin scm:checkin and scm:tag operation were still failing.

The reason for that is scm:checkin / scm:tag goals (of Maven SCM plugin) calls the version tool commands (hg commands in my case) but it was NOT passing the authentication params (username/password). For that either I could have added the and values in project pom.xml OR in ~/.m2/settings.xml withing maven-scm-plugin plugin's configuration ---OR (more secured way is to) create/use 2 new Jenkins global level variables (of password type) to create username/password variables and use / pass them to scm:checkin / scm:tag goals while calling these goals within Release plugin's configuration settings in Jenkins i.e. -Dusername=$username and -Dpassword=$password (as the values are coming from Jenkins, they will be masked automatically).

I went with the Jenkins route and create 2 password type global variables in Jenkins Global configuration under "Configure system" > Global parameters/passwords section and just passed them to scm:checkin/tag goals while calling them in Invoke Maven step (within Release plugin configuration in Jenkins).

I found, if you have ~/.hgrc set with just username in it, then Jenkins checkout /pull commands started to fail as Jenkins process stopped using the credentials what I was using to pull/clone the source code (it seems like it was giving preference to the ~/.hgrc username first as Jenkins job/process was running with jenkins user and because it didn't had a password variable/field set in ~/.hgrc, the pull/clone failed for some reason (it should have given preference to the user/credentials what we specify in the job's configuration itself). If I moved ~/.hgrc to ~/.hgrc-backup, then checkout/pull/clone worked in Jenkins worked fine (as it used the credentials what I mentioned in the Source Code Management section for Mercurial) BUT, it still failed during Maven SCM plugin not handling the under laying hg commands.

PS: running "hg push" on the workspace was working successfully (standalone, at command line) but when Maven SCM plugin was calling these goals and the goals were calling the hg commands, it didn't work for some reason.

Solution was:

  1. pass -Dusername=$username -Dpassword=$password variable to scm:checkin/scm:tag goals

  2. Make sure ~/.hgrc had username / password variables set --OR mercurial_keyring set to work with the repository (without prompting for username and password).

  3. The reason, Maven SCM plugin goals .. which called hg commands didn't work was due to an issue with the plugin I guess. A work around is to call these goals with -DpushChanges=false and this way, the goals will not call the under laying version control push operation and thus it'll succeed. Then you have to manually add another step in the "Release Plugin in Jenkins's configuration" as "Execute Shell / Execute Windows Batch command" way to run "hg push". Then, it will work and in this case, you don't need to pass -Dusername and -Dpassword parameters to these goals.

Mercurial: are pre-commit hooks run for other committing commands than “hg ci”

mercurial

Yes they do run for these operations: they run for every commit created on the local repo. You can convince yourself that they run with a simple test like: hg graft 10 --config hooks.precommit="echo 'hook ran'" It will print hook ran when the hook is called...

Mercurial get branch name by changeset

mercurial,hg-log

That's... not what revsets are for. You want to do something rather different: hg log --rev [changeset] --template "{branch}\n" See hg help templates....

create email list of all users

groovy,jenkins,jenkins-scriptler

First of all you should know that Jenkins will not always be able to tell you whether the user exists or not. From Jenkins' javadoc: This happens, for example, when the security realm is on top of the servlet implementation, there's no way of even knowing if an user of...

Booting up Vagrant from Jenkins throws Permission Denied

jenkins,vagrant,jenkins-plugins,vagrant-plugin

At last! When we try to invoke/boot Vagrant VM from Jenkins(with vagrant plugin), .vagrant.d & folder with vagrantfile will be invoked. Permission for the jenkins user should be given to these folders. In mac, right click on these folders, get info, on the bottom(permissions), click the + symbol and add...

How do I print the current classpath for a Jenkins plugin?

java,jenkins,classpath,classloader

Having read through the Javadoc for AntClassLoader it seems the solution was a lot easier than I was expecting and so I've written the following code which can be used print the classpath for a Jenkins plugin. AntClassLoader cl = (AntClassLoader) getClass().getClassLoader(); String[] classpath = cl.getClasspath().split(":"); for (String classpathElement :...

unable to get pylint output to populate the violations graph

jenkins,jenkins-plugins

it seems the correct pylint command is the following: pylint --rcfile=pylint.cfg $(find handlers -maxdepth 1 -name "*.py" -print) --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" > pylint.log || exit 0 note the addition of the --msg-template param...

Display artifact in Jenkins Build description

plugins,jenkins

You can use the Description Setter Plugin to set the build description at the end of each run. It supports variable substitution, e.g. you can use $BUILD_NUMBER. To enable HTML tags, you need to change the Markup Formatter in the Global Security options to "Raw HTML", but be aware of...

using classes in jenkins job dsl

groovy,jenkins,jenkins-job-dsl

When using println in scripts (in your example, in the runIt function), Groovy sends the call to a out variable defined in the script's binding or to System.out.println if the variable is not set. The Job DSL plugin sets this variable so that the output goes to the build log....

How can I run the Jenkins jobs through terminal?

jenkins,terminal

There are few ways to trigger a Jenkins build from command line: Remote access API is offered in a REST-like style: Job without parameters: curl -X POST JENKINS_URL/job/JOB_NAME/build --user username:token Job with parameters: curl -X POST JENKINS_URL/job/JOB_NAME/build \ --user username:token \ --data-urlencode json='{"parameter": [{"name":"id", "value":"123"}, {"name":"verbosity", "value":"high"}]}' Jenkins CLI -...

Openshift - trigger Jenkins build on git push

jenkins,openshift

You can refer to the "Configuring Which Branch to Deploy" section from the following page: https://developers.openshift.com/en/managing-deployments.html#configuring-which-branch-to-deploy...

How do I make a revset alias for tags whose names follow a pattern?

mercurial,mercurial-revsets

I tested the following that works: new($1, $2) = (::"version-" ## $2) - (::"version-" ## $1) For reference $1::$2 won't give you the same thing, it means the revision between $1 and $2 An equivalent revset that I would prefer is: new($1, $2) = only("version-" ## $2, "version-" ## $1)...

Upgrade SonarQube issues

service,jenkins,webserver,sonarqube,sonarqube-5.0

On Windows, the scripts are indeed different: if you haven't installed SonarQube as a service, you should read "Running SonarQube as a Service on Windows" to know how to start and stop if not, then: to start SonarQube, you have to execute the "StartSonar.bat" script: this will open a Command...

When does a new head is created in a repository?

mercurial

A new head can be created in three ways: * when you need to use --force on the push command, a new head is created on the repository you push to. Hint: Do never use --force (or its equivalent -f) with push when you do not have to * when...

determine if it is jenkins environment inside gradle

jenkins,gradle,jenkins-plugins,build.gradle

You could check for existence of build-specific environment variables, like $BUILD_ID or $BUILD_URL, etc. They really shouldn't exist outside of Jenkins build, but if your environment is polluted, you could have them.

Saving docker container image

jenkins,docker

Data in a Docker volume (such as /var/jenkins_home) is not preserved as part of the docker commit operation. This is intentional -- the idea is that you are persisting you data via some other mechanism, such as a host volume (-v /host/directory:/var/jenkins_home) or through the use of a data container...

Downgrade Jenkins

jenkins,downgrade

Downgrading should be equivalent to the upgrading process: To upgrade from earlier versions of Jenkins, simply redeploy the WAR file. Read this document for more about container-specific instructions on how to do this. See this document for automation. ...

Why does `su` not work in Jenkins?

shell,jenkins,su

su doesn't enter an interactive session when in a non-interactive session the way it does in an interactive session. In a shell script you get to run a single command in the su context su <user> <command>....

How to access jenkins parent job build id in a MultiJob?

maven,jenkins

I'm using this method to pass the build ID from a parent multi-job into the children jobs: In the parent job, add a "Execute Shell" build step that saves the ${BUILD_ID} into a file, like this: cat > build.properties << _EOF_ PARENT_BUILD_ID=${BUILD_ID} _EOF_ For the "MultiJob Phase" -> "Phase Jobs",...

Cut off the “desc” at N characters in hg log output with templates

mercurial,hg-log

You can use regular expressions and the sub() function to accomplish that. For example: hg log --template '{sub("^(.{0,50})(.|\n)*","\\1",desc)}\n' The sub() function takes three arguments, the pattern, the replacement, and the string to be processed. In this example, we use a group that captures anywhere from 0 to 50 characters (except...

Fail Jenkins job when nosetests fail

python,shell,jenkins,continuous-integration

I suspect nosetests will return a non zero value on failure so you can set the shell to auto fail with set -e or any of the other options in here...

Searching for the change history of partial file or path in Mercurial or TortoiseHg

mercurial,tortoisehg,file-search

I can very much advise using the hg help system for this. The most useful pages to look at (in my view): hg help revsets hg help filesets hg help patterns In the page about patterns, you can find about 'path:': To use a plain path name without any pattern...

Execute SVN Update in Jenkins - Copy a Folder to Web root Explicitly from SVN as a Build Step

powershell,batch-file,jenkins,jenkins-plugins

In an effort to help answer your question, I will explain the configuration of a job which should accommodate what you are trying to achieve: building a project under version control after an svn update has been performed and moving the generated files to a separate directory. Setup the Source...

Jenkins Flexible Publish plugin if else condition

if-statement,jenkins,jenkins-plugins

You know you can add parameters with default values under Meta Data → [ x ] This build is parameterized → Add parameter, do you? The default values are supposed to be taken if a value for a parameter isn't passed, IIRC. However, you can use the Conditional BuildStep Plugin...

Retrieving command line -D options in Java

java,maven,jenkins,jenkins-plugins

For command line options you have to use System.getProperty() instead of System.getenv(): String id = System.getProperty("classId"); Find more details here....

Single Jenkins instance using multiple Sonar instances

java,jenkins,sonarqube

Yes, in Manage Jenkins > Configure System you can add as many Sonarqube installations as you want in the Sonar section. Then when you configure a job to perform the Sonar analysis you can select what instance you want to use from a drop down list.

Excluding jobs that haven't run recently from a view in Jenkins

jenkins

Have you tried the latest version of that plugin? The following works for me to exclude old projects from a view: ...

Get the Git Working Branch in Gradle on Jenkins

git,jenkins,gradle

If this is only for testers/downstream, what about using Jenkins' Git environment variables instead, e.g. $GIT_BRANCH? Locally, it will show no version/empty when you build, but when you build on Jenkins it will show the correct branch name in your 'version' string. To get around the local builds producing an...

Jenkins (cloudbees): remotely trigger build (e.g.via instant message) w/o creating user account in Cloudbees

jenkins,cloudbees

standard token trigger mechanism requires an account in Cloudbees You can use the Build Token Root plugin to bypass authentication long enough to check the token. In the long term it would be desirable for Jenkins to let users create non-user principals that would have their own API tokens...

Advantage of Jenkins Master/Slave architecture? [on hold]

jenkins

If you have only a few jobs to run on the same platform, having only a Jenkins master to build these jobs is OK. But if you have hundreds jobs running on different platform (Linux, Windows, 32/64 bits), it's better to use Jenkins slave. You can affect some labels per...

Implementing security for users in jenkins

jenkins,jenkins-plugins

I'm using the Role Strategy plugin with a LDPA directory and it works well. You just have to install this plugin on your Jenkins server and to switch from "Matrix-based security" to "Role-Based Strategy". Next, you will be able to define roles: And assign them to users (or LDAP groups,...

can't export a variable from execute shell in Jenkins to other project (with using properties file)

bash,shell,jenkins

Did you have a look to this solution? Jenkins: How to use a variable from a pre-build shell in the Maven "Goals and options" Using a shell pre-build step + the InjectEnv plugin, you should be able to solve your problem. Update from June 22nd, I add some screen copies....

Hg Mercurial - Upgrading code not in repository

version-control,merge,mercurial,tortoisehg

Doing the merge vice versa (your changes into v3.6 might work better. Also make sure that you have selected a reasonable merge tool (internal works, but there are possibly more convenient ones out there, I use kdiff3 myself): I assume you have a repository with v3.5 and on top of...

Jenkins Console Output Error

jenkins,jenkins-plugins

I've removed the offending plug-ins (emotional jenkins and Green Spot) in an attempt to fix the problem, but I still get the errors. The stack trace shows that Green Balls is still installed. Have you followed the instructions at the top? Please check [the Jenkins] bug tracker to see...

Is it possible to save an adobe pdf file using selenium web driver and one click build Jenkins

java,pdf,selenium,jenkins

This works with Firefox: Change the Firefox profile used by Selenium (better to create a dedicated profile as described here) via Tools -> Settings -> Applications and change action of file type PDF to "Save file". In that case the window asking to open file or save will not show...

How to display all keep forever build number using Jenkins REST API for a specific job?

jenkins

This works for me: JENKINS_URL/job/JOB_NAME/api/xml?depth=2&xpath=//build[keepLog=%22true%22]/number&wrapper=forever Sample output: <forever> <number>688</number> <number>687</number> </forever> forever is a wrapper parameter. You will more details if refer to JENKINS_URL/api: For XPath that matches multiple nodes, you need to also specify the "wrapper" query parameter to specify the name of the root XML element to be...

Jenkins does not find my git repository in Visual Studio Online

git,jenkins,visual-studio-online

Well, I finally made Jenkins perform the clone of a Visual Studio Online git repository. The official Microsoft documentation is wrong. Jenkins and the Git plugin do not work as they explain there. In order to the clone operation to work, I had to put on the repository URL field...

How to force team members to comment using eclipse plugin or something else

eclipse,jenkins,eclipse-plugin,coding-style,build-automation

You can run static code checks and their corresponding eclipse plug-ins to enforce comments being made in code. For e.g. in CheckStyle javadoc can be enforced http://checkstyle.sourceforge.net/config_javadoc.html Also checkstyle can be easily integrated with Jenkins. You can also use eclipse java compiler settings for javadoc check. Go to preferences->java->compiler->javadoc to...

Triggering Jenkins to run application tests when GitHub PR is created

git,jenkins,continuous-integration,ansible,continuous-deployment

This is achievable with Jenkins. There are 3 main steps to this task: Use the Jenkins GitHub plugin to trigger a build. Use the Jenkins Ansible plugin to execute your ansible playbook during build process. Update GitHub repo with the result. This part is a little more complex since I...

Jenkins continuous deployment error

git,jenkins

You met this error because you edited the app.js file locally. So if you execute command git pull, git will pull the latest update from the remote server and as a result, this action will overwrite your local version, so git throw out the error message: error: Your local changes...

Build failure due to Sonar plugin

service,jenkins,sonarqube,sonarqube5.1

You will want to update the SonarQube Java plugin to a more recent version, your error comes from a deprecated API which was removed as of SonarQube 5.1 - and fixed in more recent versions of SonarQube Java (latest as of writing is 3.3).

Avoid large log Jenkins file (and stop build if needed)

jenkins,jenkins-plugins,diskspace

You can use the Logfilesizechecker Plugin: This plugin monitors the size of the output file of a build and aborts the build if the log file gets too big. Or, if this has also an impact on the runtime, the Build-timeout Plugin: This plugin allows you to automatically abort a...

Jenkins execute shell on job's executor during CONFIGURATION time (and access workspace)

java,shell,jenkins,jenkins-plugins

This works differently, there is no workspace for a job by default. One is allocated as soon as build is run on the build machine (be it master or a slave). There can be any number of workspaces for a given job based on where and how many times...

how to identify user who created a specific job on jenkins?

jenkins,jobs

The JobConfigHistory Plugin is ought to display a column User. And so does it in my Jenkins (v1.609.1) when I select a job's Job Config History at the bottom of the sidebar menu immediately after creating a job. (Though I agree: There's no Created there, just Changed. But, changing from...

SBT is wasting lots of disk space for dependencies each build

jenkins,sbt,playframework-2.4

It is maybe this issue: https://github.com/sbt/sbt/issues/2014? Then it should be fixed in sbt 0.13.9. See: https://twitter.com/eed3si9n/status/608309137890541569...

Why does the Jenkins SVN plugin give error E170001 when connecting to my VisualSVN server?

svn,jenkins,jenkins-plugins,visualsvn-server

In the user's Subversion folder (%APPDATA%\Subversion for Windows, ~/.subversion for Linux/Mac OS x) add http-auth-types=Basic to the global section of the servers file. Note that the case is different between Basic and basic. For me, VisualSVN reports the following list of authentication options that it supports: Negotiate NTLM Basic realm="VisualSVN...

Install gradle on Centos

jenkins,gradle

... but when I unzipped I got gradle.bat file inside bin directory which tells me that this is for Windows. It also contains a file called gradle, which is a shell script. Your download is also suitable for running on any Linux or UNIX platform .... including CentOS....

How to set up default local push path in Mercurial?

mercurial

You want to use: [paths] default = file:///home/user/myscript/ default-push = file:///home/user/myscript/ ...

How to block a Jenkins job from running at certain times

jenkins,jenkins-plugins

You can use the Conditional BuildStep plugin and set some timeframe where your job can be launched (or not). Something like that: ...

Find view name from jenkins GUI given the job details

jenkins,jenkins-plugins

That is not possible using only Jenkins UI. But you can do this with Jenkins script console. Here is a groovy sample. A job can be included in one, two or more views. So, basically, you can iterate through all views and check whether your job belongs to a concrete...

Some of my tests show prepended with junit.framework

java,unit-testing,jenkins,junit,junit4

Ok, diving deeper I realized that in all the classes I use the Assume statement. When this assume firing in the @BeforeClass, I end up with the junit.framework.TestSuite prepended. So the solution would be to avoid Assuming anything in the BeforeClass....