Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use jars from vendor/jars? #73

Open
mooreniemi opened this issue May 9, 2016 · 21 comments
Open

use jars from vendor/jars? #73

mooreniemi opened this issue May 9, 2016 · 21 comments

Comments

@mooreniemi
Copy link

I'm deploying an app in Docker. I won't have maven or the .m2 repo in that environment. I used jbundle install --vendor to put the jars in a vendor/jars directory. How do I get JBUNDLER_LOCAL_REPO to point to vendor/jars or otherwise just not point to .m2 dir?

@mooreniemi
Copy link
Author

So far what I've been doing is adding this line to the classpath.rb file but this feels hacky to me: ENV_JAVA['jars.home'] = File.expand_path( '../../vendor/jars', __FILE__ )

@mooreniemi
Copy link
Author

mooreniemi commented May 10, 2016

I also have been seeing an error in the Docker environment when I run gem install bundler, which is:

RuntimeError:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command

no such file to load -- jar/my/app/requires (LoadError)
do_require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:309
block in require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:228
require_jar_with_block at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:252
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:227
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318
block in require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:46
block in process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:69
each_line at org/jruby/RubyString.java:4764
process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:55
process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:66
require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:42
require_jars_lock! at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:194
require_jars_lock at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:215
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:226
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych_jars.rb:5
require at org/jruby/RubyKernel.java:937
(root) at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
require at org/jruby/RubyKernel.java:937
(root) at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych.rb:3
(root) at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
load_yaml at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems.rb:609
load_file at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/config_file.rb:328
initialize at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/config_file.rb:197
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/gem_runner.rb:74
load at org/jruby/RubyKernel.java:955
at /opt/jruby/jruby-9.0.5.0/bin/gem:4

@mkristian
Copy link
Owner

quick question before I look into this: do you need to use jbundler or would jar-dependencies from jruby itself do the job for you. it can use the same Jarfile but comes with a command lock_jars which produces a Jars.lock file. and vendoring works better with it.

@mooreniemi
Copy link
Author

@mkristian i'm actually not sure, to be honest. i've found it tough to sort out what my best option is. briefly, i have a gem that depends on a jar. (that much seems to be working fine) then i have an app that requires that gem, but in addition depends on another jar. the app itself i want to package as a jar ultimately. i'm happy to take your guidance on what the right practice is for my case.

@mooreniemi
Copy link
Author

mooreniemi commented May 10, 2016

Perhaps you can help me better understand how the process works? Edited: I misunderstood my error messages before, the error is happening at the gem install stage. That makes sense with the stack trace too. It looks as if doing any kind of gem install is triggering an attempt to grab my app's jar dependency, which seems wrong to me.

@mkristian
Copy link
Owner

@mooreniemi when installing a gem whether bundler or gem install a post-install hook is triggered which does look if the gems has jar dependencies and installs them. to avoid this trigger set 'JARS_SKIP=true on your shell environment. if rubocop tries to install a the gem with jar dependencies then it will trigger the same post-install hook.

this problem is with a docker container ? is it possible for you to share the Dockerfile which fails ?

packing the app as a jar - with warbler or manually ? there are many ways to pack a ruby app as a jar ;)

@mooreniemi
Copy link
Author

the post-install hook of an independent gem triggering a dependence on my project's jars seems wrong though, no? i can see it trying to trigger jars it or its transitive dependencies depend on, but that's not the case here.

this is the entirety of the script that is still failing, even with the exported env var you suggested:

#!/bin/bash
set -x
set -e

echo "STARTING STATIC ANALYSIS"

cd app_directory
export JARS_SKIP=true
jruby --version
jgem install rubocop

rubocop

i will dig up the Dockerfile for the container my app is using FROM

as for packing up into a jar, i've been using warble and that's be alright so far

@mkristian
Copy link
Owner

@mooreniemi if you use warbler you need to stick to jbundler

the post-install hook only does something on a gem with its jar dependencies. any thing else is bug. I would appreciate if can provide a reproducable case for me. or setting JARS_DEBUG=true should produce some output which can be helpful.

@mooreniemi
Copy link
Author

mooreniemi commented May 10, 2016

ill paste up the debug output soon, but fwiw i grabbed the Dockerfile

FROM java_base

RUN apk update && apk add --update bash

ENV JRUBY_VERSION 9.0.5.0
RUN mkdir -p /opt/jruby \
  && curl -fSL https://s3.amazonaws.com/jruby.org/downloads/${JRUBY_VERSION}/jruby-bin-${JRUBY_VERSION}.tar.gz -o /tmp/jruby.tar.gz \
  && tar -zxf /tmp/jruby.tar.gz -C /opt/jruby \
  && rm /tmp/jruby.tar.gz
ENV PATH /opt/jruby/jruby-${JRUBY_VERSION}/bin:$PATH

RUN echo 'gem: --no-rdoc --no-ri' >> ~/.gemrc

ENV GEM_HOME /usr/local/bundle
ENV PATH $GEM_HOME/bin:$PATH
RUN gem install bundler \
    && bundle config --global path "$GEM_HOME" \
    && bundle config --global bin "$GEM_HOME/bin"

# don't create ".bundle" in all our apps
ENV BUNDLE_APP_CONFIG $GEM_HOME

@mooreniemi
Copy link
Author

mooreniemi commented May 10, 2016

and the java_base image looks like this:

From alpine-base

RUN apk update && apk add --update wget curl
RUN apk --no-cache add ca-certificates

RUN wget --no-check-certificate https://github.com/andyshinn/alpine-pkg-glibc/releases/download/2.23-r1/glibc-2.23-r1.apk
RUN apk add --allow-untrusted glibc-2.23-r1.apk

RUN wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u65-b17/server-jre-8u65-linux-x64.tar.gz

RUN mkdir -p /usr/lib/jvm

RUN tar zxfv server-jre-8u65-linux-x64.tar.gz -C /usr/lib/jvm/
RUN mv /usr/lib/jvm/jdk1.8.0_65 /usr/lib/jvm/oracle_jdk8


ENV J2REDIR /usr/lib/jvm/oracle_jdk8
ENV PATH $PATH:/usr/lib/jvm/oracle_jdk8/bin
ENV JAVA_HOME /usr/lib/jvm/oracle_jdk8

RUN rm server-jre-8u65-linux-x64.tar.gz && \
    apk del wget ca-certificates && \
    rm -rf /tmp/* /var/cache/apk/* \
           "$JAVA_HOME"/*src.zip \
           "$JAVA_HOME"/*src.zip \
           "$JAVA_HOME"/lib/missioncontrol \
           "$JAVA_HOME"/lib/visualvm \
           "$JAVA_HOME"/lib/*javafx* \
           "$JAVA_HOME"/jre/lib/plugin.jar \
           "$JAVA_HOME"/jre/lib/ext/jfxrt.jar \
           "$JAVA_HOME"/jre/bin/javaws \
           "$JAVA_HOME"/jre/lib/javaws.jar \
           "$JAVA_HOME"/jre/lib/desktop \
           "$JAVA_HOME"/jre/plugin \
           "$JAVA_HOME"/jre/lib/deploy* \
           "$JAVA_HOME"/jre/lib/*javafx* \
           "$JAVA_HOME"/jre/lib/*jfx* \
           "$JAVA_HOME"/jre/lib/amd64/libdecora_sse.so \
           "$JAVA_HOME"/jre/lib/amd64/libprism_*.so \
           "$JAVA_HOME"/jre/lib/amd64/libfxplugins.so \
           "$JAVA_HOME"/jre/lib/amd64/libglass.so \
           "$JAVA_HOME"/jre/lib/amd64/libgstreamer-lite.so \
           "$JAVA_HOME"/jre/lib/amd64/libjavafx*.so \
           "$JAVA_HOME"/jre/lib/amd64/libjfx*.so


# Hotfix for glibc hack that fixes the order of DNS resolving (i.e. check /etc/hosts first and then lookup DNS-servers).
# To fix this we just create /etc/nsswitch.conf and add the following line:
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf

COPY ./newrelic ./newrelic
COPY ./health.sh ./health.sh
COPY ./bootstrap.sh ./bootstrap.sh

@mkristian
Copy link
Owner

thanx, will work through them tomorrow

@mooreniemi
Copy link
Author

thanks @mkristian -- i'm not seeing any additional output using the env var you suggested. i will see what i can dig up from my end.

@mooreniemi
Copy link
Author

oh this is alpine_base Dockerfile too

FROM alpine:3.3

RUN apk update && apk add --update openntpd

@mooreniemi
Copy link
Author

fyi:

i added a puts on L390 of jar_dependencies.rb:

 def require_jar( *args )
   return nil unless Jars.require?
   puts "#{args}"

and ran gem install rubocop in several different directories

  1. not my app

    C02LX2KVFD58:~/random_dir gem install rubocop
    warning: --1.9 ignored
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
    jruby: warning: unknown property jruby.cext.enabled
    ["org.yaml", "snakeyaml", "1.14"]
    ["org.bouncycastle", "bcpkix-jdk15on", "1.54"]
    ["org.bouncycastle", "bcprov-jdk15on", "1.54"]
    Successfully installed rubocop-0.40.0
    1 gem installed

  2. my app (has a Jarfile and vendor/jars)

    C02LX2KVFD58:~/my_app gem install rubocop
    warning: --1.9 ignored
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
    jruby: warning: unknown property jruby.cext.enabled
    ["org.yaml", "snakeyaml", "1.14"]
    ["custom", "internal-lib", "1.7.9"]
    ["org.javassist", "javassist", "3.19.0-GA"]
    ["com.google.code.findbugs", "annotations", "2.0.1"]
    ["log4j", "log4j", "1.2.17"]
    ["org.yaml", "snakeyaml", "1.14"]
    ["org.slf4j", "slf4j-api", "1.7.12"]
    ["commons-collections", "commons-collections", "3.2.1"]
    ["joda-time", "joda-time", "2.8.1"]
    ["commons-lang", "commons-lang", "2.6"]
    ["commons-pool", "commons-pool", "1.6"]
    ["org.reflections", "reflections", "0.9.10"]
    ["org.apache.commons", "commons-lang3", "3.0"]
    ["com.google.code.findbugs", "jsr305", "3.0.0"]
    ["commons-codec", "commons-codec", "1.10"]
    ["com.googlecode.json-simple", "json-simple", "1.1.1"]
    ["org.postgresql", "postgresql", "9.4-1201-jdbc41"]
    ["commons-validator", "commons-validator", "1.4.1"]
    ["com.rabbitmq", "amqp-client", "3.5.3"]
    ["commons-io", "commons-io", "2.4"]
    ["commons-beanutils", "commons-beanutils", "1.8.3"]
    ["com.mchange", "c3p0", "0.9.5.1"]
    ["org.jodd", "jodd-core", "3.6.6"]
    ["commons-digester", "commons-digester", "1.8.1"]
    ["org.jsoup", "jsoup", "1.8.3"]
    ["com.google.guava", "guava", "18.0"]
    ["commons-logging", "commons-logging", "1.2"]
    ["org.jodd", "jodd-upload", "3.6.6"]
    ["junit", "junit", "4.10"]
    ["javax.validation", "validation-api", "1.1.0.Final"]
    ["org.jodd", "jodd-http", "3.6.6"]
    ["org.slf4j", "slf4j-log4j12", "1.7.12"]
    ["com.mchange", "mchange-commons-java", "0.2.10"]
    ["org.hamcrest", "hamcrest-core", "1.1"]
    ["org.bouncycastle", "bcpkix-jdk15on", "1.54"]
    ["org.bouncycastle", "bcprov-jdk15on", "1.54"]
    Successfully installed rubocop-0.40.0
    1 gem installed

@mooreniemi
Copy link
Author

mooreniemi commented May 11, 2016

as further confirmation something is amiss, i changed the script being executed on my dockerized CI pipeline:

#!/bin/bash
set -x
set -e

echo "STARTING STATIC ANALYSIS"

jruby --version
export JARS_DEBUG=true
export JARS_SKIP=true
jgem install rubocop
cd app_directory
rubocop

and get the following error (the gem installs fine outside the app directory) but:

RuntimeError:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command

no such file to load -- zipcar/cheetah-sdk/1.7.9/cheetah-sdk-1.7.9 (LoadError)
do_require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:309
block in require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:228
require_jar_with_block at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:252
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:227
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318
block in require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:46
block in process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:69
each_line at org/jruby/RubyString.java:4764
process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/lock.rb:55
process at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:66
require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jars/classpath.rb:42
require_jars_lock! at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:194
require_jars_lock at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:215
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:226
require_jar at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/jar_dependencies.rb:318
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych_jars.rb:5
require at org/jruby/RubyKernel.java:937
require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych.rb:1
require at org/jruby/RubyKernel.java:937
require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/psych.rb:3
require at org/jruby/RubyKernel.java:937
require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/yaml.rb:1
require at org/jruby/RubyKernel.java:937
require at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
at /opt/jruby/jruby-9.0.5.0/lib/ruby/stdlib/yaml.rb:5
require at org/jruby/RubyKernel.java:937
(root) at /usr/local/bundle/gems/rubocop-0.40.0/lib/rubocop/config_loader_resolver.rb:1
at /usr/local/bundle/gems/rubocop-0.40.0/lib/rubocop/config_loader_resolver.rb:4
load at org/jruby/RubyKernel.java:955
at /usr/local/bundle/bin/rubocop:23

@mkristian
Copy link
Owner

@mooreniemi this looks ok for me. installing gems need yaml and yaml needs psych and psych needs a jar and then all jars gets installed when there is Jars.lock file. is there such file inside the vendor/jars directory ?

one thing I was thinking about: the error with you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command means there jar-dependencies can find the jars.

I thought you have the jars vendored ?

@mooreniemi
Copy link
Author

@mkristian i'm really confused. if i am installing rubocop -- i expect that to install the same in any directory. why would it depend on directory whether it loads extraneous jars it doesn't even use? this is precisely what i don't want in a dep manager: i want to deterministically know i'm loading stuff i need. "and then all jars gets installed" is just a huge leap to me.

i do have the jars vendored, but that was my original question: it never seems to pick that up, and always tries to go for .m2/ if i could force my project to use the jars in vendor/jars this problem would be papered over, but i still think it is bad behavior

@mkristian
Copy link
Owner

export JARS_HOME=path/to/vendor/jars - not sure whether this what you tried.

since I do not fully understand why you see all those jars loaded, I can not say it is intended or not. is there a Jars.lock file some where in app_directory or app_directory/vendor or app_directory/vendor/jars ?

@mooreniemi
Copy link
Author

vendor/jars contains a Jars.lock. i will try exporting that env var for my docker containers. i bet it will fix this, but i'd still say just calling gem or other ruby executables that require a jar shouldnt pull in all jars if, by chance, some other Jars.lock file exists

@mkristian
Copy link
Owner

looking into those vendor dirs is too much, you convinced me. ignoring the Jars.lock in the current directory - not sure. so you rather have the application call Jars.setup or something ? I appreciate your input !

@mooreniemi
Copy link
Author

exporting of the env var resolved my issue for the moment. maybe i could PR some wiki/readme changes to help the next peeps along?

i do think i would rather the application call Jars.setup or like lock_jar jars = LockJar.load. do you know how bundler handles this generally? they might be good to model off of, i don't know in depth :/

thanks for all your responses by the way, really appreciate the work you do on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants