Jenkins - java.lang.IllegalArgumentException: Last

2020-02-02 03:49发布

站内文章 / 移动开发
543 0 0

问题:

Jenkins 2.176.4-3 rolling

Gradle 4.3.1

Issue area: Parallel run of a given single Gradle task (or it could be any simple action) and especially when running concurrent runs of Jenkinsfile based pipelines

All the sudden I got this error in Jenkins log page, never seen this error before (found no stackoverflow posts either for this error in Jenkins).

Error: java.lang.IllegalArgumentException: Last unit does not have enough valid bits

  1. For some reason the previous build failed and automatically took me to Jenkins logs page showing Jenkins angry fire face:

  2. Re-ran just one run of Jenkinsfile pipeline again (which runs a single Gradle task (lets say: rpm task) present in multiple projects in parallel) and it worked successfully!!; checked parameters/values passed between successful and failed runs - they were similar.

Example Jenkinsfile can be found here: Jenkinsfile Pipeline DSL: How to Show Multi-Columns in Jobs dashboard GUI - For all Dynamically created stages - When within PIPELINE section where I'm running a gradle task (lets say gradle rpm) in parallel (each gradle rpm task running in individual stage step) by creating dynamic stages (so that I don't hard-code them in Jenkinsfile for each project).

NOTE: In Jenkinsfile, even though the gradle task rpm (that I'm running in parallel in individual stage (under stages) section has a valid gradle task defined i.e. rpm, it seems like Gradle task running on multiple projects in parallel (section within Jenkinsfile) is not 100% successful everytime, with at least Gradle 4.3.1 as I'm also seeing this other error: 

05:31:41 FAILURE: Build failed with an exception.
05:31:42 
05:31:42 * What went wrong:
05:31:42 Task 'null' not found in root project 'T000123_ABCD_Project'.
05:31:42

This issue poster said similar here: gradle issue after distribution 4.10.2


Here is the Full error log in Jenkins stack trace is showing:

 Oops!

A problem occurred while processing the request.
        Please check our bug tracker to see if a similar problem has already been reported.
        If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem.
        If you think this is a new issue, please file a new issue.
        When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins.
        The users list might be also useful in understanding what has happened.
Stack trace

java.lang.IllegalArgumentException: Last unit does not have enough valid bits
    at java.util.Base64$Decoder.decode0(Base64.java:734)
    at java.util.Base64$Decoder.decode(Base64.java:526)
    at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:41)
    at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:53)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.console.ConsoleNote.readFrom(ConsoleNote.java:258)
    at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:111)
    at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
    at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
    at org.jenkinsci.plugins.workflow.log.FileLogStorage$1$1.write(FileLogStorage.java:238)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
    at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
    at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:314)
    at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:229)
    at hudson.console.AnnotatedLargeText.writeRawLogTo(AnnotatedLargeText.java:168)
    at org.jenkinsci.plugins.workflow.log.FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)
    at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:147)
    at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:264)
    at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:95)
    at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
Caused: javax.servlet.ServletException
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:797)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
    at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
    at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:456)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at com.cloudbees.jenkins.support.impl.cloudbees.UnrestrictedApiCallsMonitor$ApiMonitorFilter.doFilter(UnrestrictedApiCallsMonitor.java:120)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:502)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
    at java.lang.Thread.run(Thread.java:745)


What's making me get this error message in Jenkins?

Error: java.lang.IllegalArgumentException: Last unit does not have enough valid bits

and


Why my pipeline runs fail intermittently with the following error message in Gradle (when parallel section is used to run something (i.e. a Gradle someTask) within Jenkinsfile)?

Task 'null' not found in root project



When running many concurrent runs of this pipeline (which runs a Gradle task on many projects in parallel), let's say I launch 4 pipelines runs and each is building 3-5 project's Gradle rpm task), then I saw the following issues related to FILE locking.

https://stackoverflow.com/a/59828842/1499296

i.e. Gradle not able to delete folder/file (as listed below)

**Execution failed for task ':clean'.
> Unable to delete directory: /some/path/build/someFolder/aFolder_or_File **

or

Failed to capture snapshot of output files for task due to Timeout waiting to lock file hash cache

Failed to capture snapshot of output files for task ':rpm' property 'archivePath' during up-to-date check.

> Timeout waiting to lock file hash cache (/view/user123456_Team_Tools/vobs/space/test/folder1/Project1of5/.gradle/4.3.1/fileHashes). It is currently in use by another Gradle instance.

  Owner PID: 29003

  Our PID: 903

  Owner Operation: 

  Our operation: 

  Lock file: /view/user123456_Team_Tools/vobs/space/test/folder1/Project1of5/.gradle/4.3.1/fileHashes/fileHashes.lock

or

getting: Could not create service of type ScriptPluginFactory using BuildScopeServices.createScriptPluginFactory()

17:40:11 FAILURE: Build failed with an exception.
17:40:11 
17:40:11 * What went wrong:
17:40:11 Could not create service of type ScriptPluginFactory using BuildScopeServices.createScriptPluginFactory().
17:40:11 > Could not create service of type FileHasher using BuildSessionScopeServices.createFileSnapshotter().

or

Could not read path '/some/jenkins/workspace/path/of/a/job/folder/sub-folder/file.class_or_any_extension'

or

java.io.FileNotFoundException i.e. Not able to find a file (that Gradle is builds successfully if I run only one run of this pipeline).

17:38:28 * What went wrong:
17:38:28 Execution failed for task ':rpm'.
17:38:28 > java.io.FileNotFoundException: '/some/jenkins/workspace/path/of/a/job/folder/sub-folder/artifact-1.0.0.121.rpm'

回答1:

At a single pipeline run case, if you have even more projects (let's say 20+ projects) and you are running Gradle someTask in parallel (whether in a single stage or individual stages per project), then you could set GRADLE_USER_HOME=$WORKSPACE/${whichProject} folder (where ${whichProject} is a project folder which will be available to you in Jenkins WORKSPACE (after git clone or similar checkout operation)). This could possibly solve any parallel task running into any locking related issues but at the cost of compromising Gradle's performance (i.e. for not using Gradle CACHE among many project's parallel task run).

Setting a unique GRADLE_USER_HOME value (per project) inside dynamically created unique Jenkins workspace (example: /jenkins/workspaces/jobname/<buildnumber>/ where BUILD_NUMBER makes the Jenkins workspace folder unique) will resolve any lock related issues

  1. i.e. Doing this, Project1 will use GRADLE_USER_HOME=/jenkins/workspaces/jobname/<buildnumber>/Project1 (and now .gradle and caches folders will sit under this folder and used by Gradle and won't collide with other project's Gradle CACHE folder: .gradle and caches which are also running in parallel).

I noticed CLI option --gradle-user-home (aka $GRADLE_USER_HOME) while running Gradle task was getting set to just Jenkins's WORKSPACE example value only and thus, all 3-5 project's Gradle task (in my case rpm task) runs were running in parallel (using the same service account's Gradle caches folder) which usually sits under GRADLE_USER_HOME location (so in my case single Gradle's caches folder was getting shared by all those 3-5 projects for parallel task run).

I'm still trying to find the solution for When running concurrent pipeline runs (which again runs parallel Gradle rpm task per project within a single pipeline run) each using their own different Jenkins WORKSPACE folders, why I'm still getting these issues; I'll update if I find a solution.



标签： jenkins gradle parallel-processing locking jenkins-pipeline
举报

收藏的人(0)

Ta的文章 更多文章
登录 后发表评论
0条评论
还没有人评论过~