torstai 8. syyskuuta 2016

Getting full error message from "docker service ps"


I was trying out docker swarm, network and services, and for some reason my nginx containers failed to start. Unfortunately, "docker service ps my-web" truncated the error, giving something like below
e5qw27qr4qbc9vrm68g3i9tl0   my-web.1  nginx  node3  Shutdown       Failed 2 seconds ago          "starting container failed: ca…"
There will be "--no-trunc" in version 1.13, which should resolve this. Meanwhile, using "docker inspect e5qw27qr4qbc9vrm68g3i9tl0" (id from docker service ps) gave the full error message.

In this case, the VM created with docker-machine did not have necessary pieces to connect secured network.

sunnuntai 7. elokuuta 2016

Ubuntu Xenial64 on Virtual Box and Vagrant


There was a lot of strange problems with ubuntu/xenial64, and in https://github.com/mitchellh/vagrant/issues/6616 there is a mention by Seth Vargo (employee of Hashicorp)
    
The ubuntu/xenial64 box is built wrong and horribly broken. Please note that "ubuntu" is the name of a user, not a representation of a canonical source for ubuntu images. Please try bento/ubuntu-16.04 instead. Thanks.
https://github.com/mitchellh/vagrant/issues/6616#issuecomment-227776489

These errors included following:

rejecting i/o to offline device
This happened almost everytime after heavier I/O operations, for example after loading Docker images.

stderr: Inappropriate ioctl for device
I think that this happened when Vagrant tried to setup network interfaces, mainly "enp0s8".

So just use bento/ubuntu-16.04

tiistai 5. heinäkuuta 2016

Jenkins Workflow: Executing build step for every change in commit

At work, we wanted send an email for every change that was made in a project. By default, Jenkins likes to collate changes into as few builds as possible, and normally sends an email per build.

The solution seemed to be usage of Jenkins Pipeline. Jenkins Pipeline enables creation and execution of jobs "on the fly" as needed.

First problem was to get access to ChangeLogSet. There is some preset variables in Jenkinsfile, but I could not find documentation for them. After some googling, Stack Overflow came to rescue.

def changes = currentBuild.rawBuild.changeSets

But when this was executed, Jenkins complained
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild


There's a "In-process Script Approval" -tool in Jenkins, where you can allow usage of these methods.

After that was solved, next problem was with serialization. As the actual job execution is transferred to different node, every non-serializable object caused an exception. To prevent this, I had to null objects in proper places. This then prevented running jobs in loop, as the variables in loops needed to be nulled before job execution. So I had to collect jobs into map, and after every job was defined, null everything and use "parallel" -task to execute jobs.

So the whole thing is here:

//changes is http://javadoc.jenkins-ci.org/hudson/scm/ChangeLogSet.html
def changes = currentBuild.rawBuild.changeSets
//We need to create branches for later execution, as otherwise there would be serialization exceptions
branches = [:]
for (int j = 0; j < changes.size(); j++) {
    def change = changes.get(j)
    for (int i = 0; i < change.getItems().size(); i++) {
        def entry = change.getItems()[i]
        def commitTitleWithCaseNumber = entry.getMsg()
        def commitMessage = entry.getComment()
        //split from first non digit
        def caseNumber = (commitTitleWithCaseNumber =~ /^[0-9]*/)
        // check that caseNumber was in case place
        if( !caseNumber[0].isEmpty() && commitTitleWithCaseNumber.startsWith(caseNumber[0])) {
          // Remove number from title, just for nicer subject line
          def commitTitle = commitTitleWithCaseNumber.substring(caseNumber[0].length()).trim()
          def number = caseNumber[0]
          branches["mail-${j}-${i}"] = {
              node {
                  emailext body: commitMessage, subject: "[Sysart ${number}] ${commitTitle}", to: 'redacted@example.com'
             }
          }
        }
        // Need to forcibly null all non serializable classes
        caseNumber = null
        entry = null
    }
    change = null
}
changes = null
stage 'Mail'
parallel branches
This was a little more difficult that I had expected, mainly because of serialization complications. But in the end, it works so it cannot be completely stupid.

maanantai 27. kesäkuuta 2016

docker: Error response from daemon: invalid bit range [4, 4]

Fooling around with docker, trying to create a overlay network. Copied some settings from net, and when starting container, docker reported an error.

root@infra-front:~# docker network create -d overlay --subnet=192.168.50.0/24 --gateway=192.168.50.1 --ip-range=192.168.50.4/32 test

32bdd738a6b7444d8c9a471e451793acd8793db1739f4f660b9981df018252c1

root@infra-front:~# docker run --rm -ti --net test alpine sh

docker: Error response from daemon: invalid bit range [4, 4].

It seems that my network settings where wrong. For now, I just removed gateway and subnet and things started to work.

tiistai 14. kesäkuuta 2016

Jaspersoft Studio 6.2.2 on Fedora 23: no swt-pi-gtk in java.library.path

When starting Jaspersoft Studio 6.2.2 only thing I got was

Jaspersoft Studio:
GTK+ Version Check
Jaspersoft Studio:
An error has occurred. See the log file
/home/jyrki/projects/jasper/TIBCOJaspersoftStudio-6.2.2.final/configuration/1465956968426.log.

Log file had:

TIBCOJaspersoftStudio-6.2.2.final/configuration/org.eclipse.osgi/264/0/.cp/libswt-pi-gtk-4530.so: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory
no swt-pi-gtk in java.library.path
/home/jyrki/.swt/lib/linux/x86/libswt-pi-gtk-4530.so: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory
Can't load library: /home/jyrki/.swt/lib/linux/x86/libswt-pi-gtk.so
Problem got fixed after installing gtk2.i686 (32 bit version)

sudo dnf install gtk2.i686

Using ldd (print shared library dependencies) helped to find out what was actually missing, as the error message is somewhat miss leading (Can't load library: /home/jyrki/.swt/lib/linux/x86/libswt-pi-gtk.so)

ldd /home/jyrki/projects/jasper/TIBCOJaspersoftStudio-6.2.2.final/configuration/org.eclipse.osgi/264/0/.cp/libswt-pi-gtk-4530.so
ldd: warning: you do not have execution permission for `/home/jyrki/projects/jasper/TIBCOJaspersoftStudio-6.2.2.final/configuration/org.eclipse.osgi/264/0/.cp/libswt-pi-gtk-4530.so'
linux-gate.so.1 (0xf7741000)
libgtk-x11-2.0.so.0 => not found
libgthread-2.0.so.0 => /lib/libgthread-2.0.so.0 (0xf76af000)
libXtst.so.6 => /lib/libXtst.so.6 (0xf76a8000)
libc.so.6 => /lib/libc.so.6 (0xf74da000)
libpthread.so.0 => /lib/libpthread.so.0 (0xf74bd000)
libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0xf737b000)
libX11.so.6 => /lib/libX11.so.6 (0xf723a000)
libXext.so.6 => /lib/libXext.so.6 (0xf7226000)
libXi.so.6 => /lib/libXi.so.6 (0xf7214000)
/lib/ld-linux.so.2 (0x5660d000)
libxcb.so.1 => /lib/libxcb.so.1 (0xf71ed000)
libdl.so.2 => /lib/libdl.so.2 (0xf71e8000)
libXau.so.6 => /lib/libXau.so.6 (0xf71e4000)

tiistai 19. huhtikuuta 2016

Using Keycloak APIs: "RESTEASY004655: Unable to invoke request"


Following exception was thrown while executing multiple calls to Keycloak API.


Caused by: javax.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request
at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:436)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:102)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:64)
at com.sun.proxy.$Proxy276.findAll(Unknown Source)
at org.keycloak.admin.client.resource.ClientsResource$findAll.call(Unknown Source)
Caused by: java.lang.IllegalStateException: Invalid use of BasicClientConnManager: connection still allocated.
Make sure to release the connection before allocating another one.
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.impl.conn.BasicClientConnectionManager.getConnection(BasicClientConnectionManager.java:160)
at org.apache.http.impl.conn.BasicClientConnectionManager$1.getConnection(BasicClientConnectionManager.java:142)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)

I was calling

keycloak.realm(realm).clients().create(representation) 
and did not read anything from response. Simple fix was

            def response = keycloak.realm(realm).clients().create(representation)
            response.close()


lauantai 26. maaliskuuta 2016

Problem with Kubernetes SkyDNS healtz

I had some problems when trying to get DNS working on Kubernetes. I followed instructions from https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns. Everything seemed to be working well, but the pod got restarted after 30 seconds. The log for healthz -container had following entries:

2016/03/19 04:25:25 Client ip 10.0.96.1:50326 requesting /healthz probe servicing cmd sleep 10 && nslookup kubernetes.default.svc.kube.local localhost >/dev/null
2016/03/19 04:25:25 Healthz probe error: Result of last exec: nslookup: can't resolve 'kubernetes.default.svc.kube.local': Name does not resolve, at 2016-03-19 04:25:23.967737423 +0000 UTC, error exit status 1
After trying a lot of things, I found a bug report for Alpine Linux. Basically, the nslookup does not respect the server parameter, if /etc/resolv.conf  has entries. Comment on that issues recommends using dig or drill for querying.

So I made a simple image and pushed it into docker hub, https://hub.docker.com/r/jyrki/arm-kubernetes-healthz-drill/. Nothing fancy, just added drill (https://github.com/jyrkiput/arm-kubernetes-healthz-drill/blob/master/Dockerfile). I used the existing image as base as I wanted to have the exechealtz available.

Then I had to change the healtz command to
drill -q kubernetes.default.svc.kube.local @localhost