Month: October 2020

After A LOT of struggling to get Nexus 3 running behind Traefik2 I finally got in working, so I thought let’s share this with rest of the world… 😀

Running the GUI behind Traefik2 wasn’t a big deal… It was logging in and pushing the images to it that was a pain in the ass…

So… For everyone that is struggling with the same issue… Here is the answer… (I hope for you). And the problem wasn’t even my docker-compose file… But a setting IN Nexus 3…

My Setup is a docker swarm with 5 nodes. So just keep in mind that my docker-compose file is for a swarm (includes deploy settings and stuff). As an extra I also run my persistent storage on NFS. So it doesn’t matter on which worker the conatiner get’s deployed… So let’s start with the compose files:

Traefik V2

version: "3.7"
networks:
  proxy:
    driver: overlay
    external: true
  default:
    driver: overlay

########################### SERVICES
services:
  ############################# FRONTENDS

  # Traefik 2 - Reverse Proxy
  # Touch (create empty files) traefik.log and acme/acme.json. Set acme.json permissions to 600.
  # touch $DOCKERDIR/traefik2/acme/acme.json
  # chmod 600 $DOCKERDIR/traefik2/acme/acme.json
  # touch $DOCKERDIR/traefik2/traefik.log
  traefik:
    image: traefik:v2.2
    environment:
      - AWS_HOSTED_ZONE_ID=${AWS_HOSTED_ZONE_ID}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    hostname: traefik
    ports:
      - "80:80"
      - "443:443"
    deploy:
      restart_policy:
         condition: on-failure
      mode: replicated
      placement:
        constraints:
        - node.role == manager
        - node.hostname == elb.mydomain.com
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=proxy"
        - "traefik.http.routers.api.entrypoints=https"
        - "traefik.http.routers.api.rule=Host(`elb.mydomain.com`)"
        - "traefik.http.routers.api.service=api@internal"
        - "traefik.http.routers.api.tls=true"
        - "traefik.http.routers.api.tls.domains[0].main=mydomain.com"
        - "traefik.http.routers.api.tls.domains[0].sans=*.mydomain.com"
        - "traefik.http.routers.api.tls.certresolver=mytlschallenge"
        - "traefik.http.routers.api_http.entrypoints=http"
        - "traefik.http.routers.api_http.rule=Host(`elb.mydomain.com`)"
        - "traefik.http.routers.api_http.middlewares=traefik-redirectscheme"
        - "traefik.http.middlewares.traefik-redirectscheme.redirectscheme.scheme=https"
        - "traefik.http.services.api.loadbalancer.server.port=8080"
        ## Middlewares
        - "traefik.http.routers.traefik-rtr.middlewares=middlewares-basic-auth@file"
    command:
      - --api.insecure=true # set to 'false' on production
      - --api.dashboard=true
      - --api.debug=false
      - --log.level=WARN
      - --providers.docker=true
      - --providers.docker.swarmMode=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.network=proxy
      - --entrypoints.http.address=:80
      - --entrypoints.https.address=:443
      - --certificatesresolvers.mytlschallenge.acme.dnsChallenge.resolvers=1.1.1.1:53,8.8.8.8:53
      #- --certificatesResolvers.mytlschallenge.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory # Generates LE test certificates.  Can be removed for production
      - --certificatesResolvers.mytlschallenge.acme.dnsChallenge=true
      - --certificatesResolvers.mytlschallenge.acme.dnsChallenge.provider=route53
      - --certificatesresolvers.mytlschallenge.acme.email=${LE_EMAIL}
      - --certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json
      - --serverstransport.insecureskipverify=true

    volumes:
      - "nfs_traefik:/letsencrypt"
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy
volumes:
  nfs_traefik:
    external: true

Nexus 3:

version: "3.7"
services:
  nexus:
    image: sonatype/nexus3
    environment:
      - "REGISTRY_HTTP_RELATIVEURLS=true"
      - "TZ=Europe/Brussels"
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
         condition: on-failure
      placement:
        constraints:
          - node.role == worker
      labels:
        - "traefik.enable=true"
        # Nexus Interface
        - "traefik.http.routers.nexus.entrypoints=https"
        - "traefik.http.routers.nexus.service=nexus"
        - "traefik.http.routers.nexus.rule=Host(`nexus.mydomain.com`)"
        - "traefik.http.routers.nexus.tls.certresolver=mytlschallenge"
        - "traefik.http.services.nexus.loadbalancer.server.port=8081"
        # Regsitry Endpoint
        - "traefik.http.routers.registry.rule=Host(`registry.mydomain.com`)"
        - "traefik.http.routers.registry.tls=true"
        - "traefik.http.routers.registry.service=registry"
        - "traefik.http.routers.registry.tls.certresolver=mytlschallenge"
        - "traefik.http.services.registry.loadbalancer.server.port=5000"
        - "traefik.docker.network=proxy"
    volumes:
      - "nexus_data:/nexus-data"
    networks:
      - proxy
    ports:
      - "8081:8081"
      - "5000:5000"

volumes:
  nexus_data:
    driver: local
    driver_opts:
      type: nfs
      o: addr=192.168.86.12,rw
      device: ":/volume1/Docker/NexusData"
networks:
  proxy:
    external: true

If you deploy these 2 compose files (replacing mydomain.com with your domain) you should have a running traefik V2 (with GUI, on elb.mydomain.com), a nexus running on nexus.mydomain.com and a repository endpoint on registry.mydomain.com.

The first time you open nexus you will be asked to give the admin user and the password… You can find this password under /nexus-data/admin.password in your container or nfs share if you did the same as me. Afterwards, just follow the setup.

• 

• 

Let’s create a Docker repository
Go to –> Settings –> Repositories –> Create Repository

• Give the name of your repository
• Check the HTTP connector and add port 5000 (the port you mentioned in the traefik labels for the repository url)
• If you want to allow anonymous pulls, you can check that too (optional)
• And I’ve also enabled the Docker V1 api (optional)
• Click on “Save”

Now, if you try to login…

koen@pop-os:~/Projects/Docker/docker-ha/build$ docker login -u admin -p Password https://registry.mydomain.com:5000
Error response from daemon: Get https://registry.mydomain.com:5000/v2/: dial tcp 192.168.86.200:5000: connect: connection refused
OR
koen@pop-os:~/Projects/Docker/docker-ha/build$ docker login -u admin -p Password https://registry.mydomain.com
Error response from daemon: login attempt to https://registry.mydomain.com/v2/ failed with status: 404 Not Found

After A LOT of Googling around… I finally found the solution to this problem… Which wasn’t the traefik config… So here it comes… 😀

Go to Settings –> Realms and add the Docker Bearer Token Realm…

Now try to login again…

koen@pop-os:~/Projects/Docker/docker-ha/build$ docker login -u admin -p Password https://registry.mydomain.com

Login Succeeded