chipster-tech — technical discussion

Re: [Chipster-tech] Login Page Fails to Initialise

[Regan C. <reg...@gm...>]

Please excuse my previous email. I simply needed to clear the cache and
cookies from a previous installation of Chipster.

The login page initialises and the login works (see screenshots).

Kind regards
Regan Cannell

On Thu, Aug 1, 2024 at 1:40 PM Regan Cannell <reg...@gm...> wrote:

> To whom it may concern
>
> I've deployed an instance of Chipster on a Linux machine (Ubuntu 22.04)
> using `k3s`. All the pods are up and running. However, the login page fails
> to initialise (please see attached screenshots). Please give me some advice
> on what services, pods, or config maps to investigate to solve this problem.
>
> Kind regards
> Regan Cannell
>

[Chipster-tech] Login Page Fails to Initialise

[Regan C. <reg...@gm...>]

To whom it may concern

I've deployed an instance of Chipster on a Linux machine (Ubuntu 22.04)
using `k3s`. All the pods are up and running. However, the login page fails
to initialise (please see attached screenshots). Please give me some advice
on what services, pods, or config maps to investigate to solve this problem.

Kind regards
Regan Cannell

Re: [Chipster-tech] Running chipster in Kubernetes

[Lukáš H. <xhe...@ic...>]

Yes, thanks, it is working now.

> On 1. 2. 2024, at 9:51, chi...@cs... wrote:
> 
> Hi Lukáš,
> 
> I assume this came from tool "Extract .gz file". This was fixed in our development versions a couple weeks ago and I have just made a new release version which includes this fix.
> 
> Please update to version v4.9.8 (https://github.com/chipster/chipster-openshift/blob/k3s/k3s/README.md#updates) and it should work again.
> 
> Bug description: The file tool_utils.py is used in many Python tools e.g. to handle filenames. At some point we added code there, which works only in Python 3. However, some tools are still stuck in Python 2, but luckily those don't need tool_utils.py. This tool, gunzip.py was still running in Python 2 and also needs tool_utils.py. Luckily in this case the fix was easy, gunzip.py works just fine in Python 3 too (https://github.com/chipster/chipster-tools/commit/db26333514c0891beaa4e8a46699aff7d1a39ae0).
> 
> Thanks for reporting this.
> 
> Best regards,
> Petri
> 
> 
> ----- Original Message -----
> From: "Lukáš Hejtmánek via Chipster-tech" <chi...@li...>
> To: chi...@li...
> Cc: "Lukáš Hejtmánek" <xhe...@ic...>
> Sent: Wednesday, 31 January, 2024 17:12:45
> Subject: Re: [Chipster-tech] Running chipster in Kubernetes
> 
> Hello,
> 
> we encountered another bug in tools:
> (most recent call last):
> File "<stdin>", line 23, in <module>
> File "/opt/chipster/jobs-data/b577e0ae-d149-4121-afeb-74c1e3309ab8/7cbfcc58-6ae2-4ccf-aacd-ccf719ff2d24/data/../toolbox/tools/common/python/lib/tool_utils.py", line 37
> return f"{num:3.1f} {unit}{suffix}"
> ^
> SyntaxError: invalid syntax
> 
> 
> is there anything we can do?
> 
> --
> Lukáš Hejtmánek
> 
> Linux Administrator only because
>  Full Time Multitasking Ninja 
>  is not an official job title
> 
> 
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chipster-tech

--
Lukáš Hejtmánek

Linux Administrator only because
  Full Time Multitasking Ninja 
  is not an official job title

Re: [Chipster-tech] Running chipster in Kubernetes

[<chi...@cs...>]

Hi Lukáš,

I assume this came from tool "Extract .gz file". This was fixed in our development versions a couple weeks ago and I have just made a new release version which includes this fix.

Please update to version v4.9.8 (https://github.com/chipster/chipster-openshift/blob/k3s/k3s/README.md#updates) and it should work again.

Bug description: The file tool_utils.py is used in many Python tools e.g. to handle filenames. At some point we added code there, which works only in Python 3. However, some tools are still stuck in Python 2, but luckily those don't need tool_utils.py. This tool, gunzip.py was still running in Python 2 and also needs tool_utils.py. Luckily in this case the fix was easy, gunzip.py works just fine in Python 3 too (https://github.com/chipster/chipster-tools/commit/db26333514c0891beaa4e8a46699aff7d1a39ae0).

Thanks for reporting this.

Best regards,
Petri


----- Original Message -----
From: "Lukáš Hejtmánek via Chipster-tech" <chi...@li...>
To: chi...@li...
Cc: "Lukáš Hejtmánek" <xhe...@ic...>
Sent: Wednesday, 31 January, 2024 17:12:45
Subject: Re: [Chipster-tech] Running chipster in Kubernetes

Hello,

we encountered another bug in tools:
(most recent call last):
File "<stdin>", line 23, in <module>
File "/opt/chipster/jobs-data/b577e0ae-d149-4121-afeb-74c1e3309ab8/7cbfcc58-6ae2-4ccf-aacd-ccf719ff2d24/data/../toolbox/tools/common/python/lib/tool_utils.py", line 37
return f"{num:3.1f} {unit}{suffix}"
^
SyntaxError: invalid syntax


is there anything we can do?

--
Lukáš Hejtmánek

Linux Administrator only because
  Full Time Multitasking Ninja 
  is not an official job title


_______________________________________________
Chipster-tech mailing list
Chi...@li...
https://lists.sourceforge.net/lists/listinfo/chipster-tech

Re: [Chipster-tech] Running chipster in Kubernetes

[Lukáš H. <xhe...@ic...>]

Hello,

we encountered another bug in tools:
(most recent call last):
File "<stdin>", line 23, in <module>
File "/opt/chipster/jobs-data/b577e0ae-d149-4121-afeb-74c1e3309ab8/7cbfcc58-6ae2-4ccf-aacd-ccf719ff2d24/data/../toolbox/tools/common/python/lib/tool_utils.py", line 37
return f"{num:3.1f} {unit}{suffix}"
^
SyntaxError: invalid syntax


is there anything we can do?

--
Lukáš Hejtmánek

Linux Administrator only because
  Full Time Multitasking Ninja 
  is not an official job title

Re: [Chipster-tech] [Extern] - Re: Microarray PCA analysis not starting

[Oliver H. <o....@dk...>]

Hi Eija,

it's only about changes in microarray tools.

I think there are no changes needed in any other tools. If I remember 
correctly
I searched (vie grep) through all tools and only found relevant changes 
in microarrays.

The motivation for the changes have been, that people used some 
characters in their
phenodata (was it sample names? or group names?) which made reading them in
to fail in the tools R scripts.

As these are not really clear bugs I did not made an PR before. But now, 
with the current
updates those changes did cause some headaches on my side :-) So I would 
be happy
if those could go into main/master some day.

Well, I think, it's best to wait for Petri or somebody who is willing to 
check if those changes
make sense and improve the tools in general.

All changes can be viewed already here:

https://github.com/chipster/chipster-tools/compare/master...oheil:chipster-tools:dkfz-tools

Best regards,

Oli



Am 11.09.2023 um 09:10 schrieb chi...@cs...:
> Hi Oliver,
>
> Petri is out of office so just a quick note that yes, it might take 
> some weeks until we could start a merge discussion. Is the phenodata 
> line mainly in microarray analysis tools or have you replaced it in 
> every tool?
>
> Best,
> Eija
>
> ------------------------------------------------------------------------
> *From: *"Oliver Heil" <o....@dk...>
> *To: *"chipster-tech" <chi...@li...>
> *Cc: *"Konrad, Aline" <a.k...@dk...>
> *Sent: *Friday, 8 September, 2023 13:14:41
> *Subject: *Re: [Chipster-tech] [Extern] - Re: Microarray PCA analysis 
> not starting
>
> Hi Petri,
>
>     Or if you have your own git branch (check with 'git status'):
>
>     cd ~/git/chipster-tools/
>     git switch master
>     git pull
>     git switch <your-own-branch>
>     git merge master
>     bash reload.bash
>
> This.
>
> I have my own branch 'dkfz-tools' here
> https://github.com/oheil/chipster-tools
>
> I did as you said and it seems to be fine now.
>
> I am now in sync with master on master and ahead 3 commits on dkfz-tools
> Most changes in branch dkfz-tools I could make a PR, it's mainly this:
>
>     phenodata <- read.table('phenodata.tsv', header=TRUE, sep='\t',
>     quote='', as.is=TRUE, check.names=FALSE, comment.char='')
>
> instead of
>
>     phenodata <- read.table("phenodata.tsv", header = TRUE, sep =
>     "\t", as.is = TRUE)
>
> But there are also some changes, e.g. for special chip types, which 
> you may not want.
>
> Do you want me to do a PR from dkfz-tools now?
> My guess is, that it wouldn't be merged right away but that we would 
> go into
> a few discussions, which is not a problem on my side. But if you say, 
> it may take weeks
> until we start a merge discussion I would say, ok, lets do it at some 
> time in future...
>
> Regards,
>
> Oli
>
>
> Am 08.09.2023 um 07:02 schrieb Chipster:
>
>     Hi Oli,
>
>     interesting. It looks like you still have the old tool scripts
>     somehow. Have you mounted the tool scripts from the host
>     (https://github.com/chipster/chipster-openshift/blob/master/k3s/tool-script-dev.md#mount-tool-scripts-from-the-host)
>     , i.e. something like this in your values.yaml?
>
>       tools:
>         hostPath: /opt/chipster/toolbox/tools
>
>     In that case you would have to also update the tool scripts on
>     your host:
>
>     cd ~/git/chipster-tools/
>     git pull
>     bash reload.bash
>
>     Unfortunately the reload script hasn't been very reliable. It
>     should print some logs. If the log timestamps are not from the
>     current UTC time, then the reload didn't happen. In that case,
>     restart the whole toolbox and wait for the old pod to disappear:
>
>     kubectl rollout restart deployment/toolbox
>     watch kubectl get pod
>
>     Or if you have your own git branch (check with 'git status'):
>
>     cd ~/git/chipster-tools/
>     git switch master
>     git pull
>     git switch <your-own-branch>
>     git merge master
>     bash reload.bash
>
>     If you haven't mounted the tools scripts from the host, then the
>     tool scripts should come directly from the toolbox container image
>     and the version tag should make sure you are running the latest
>     version of it. I guess we should then check how to toolbox
>     deployment looks like:
>
>     kubectl get deployment/toolbox -o yaml
>
>     Best regards,
>     Petri
>
> -- 
>
> Oliver Heil
> Microarray Core Facility
> Bioinformatics
>
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
>
> o....@dk... <mailto:o....@dk...>
> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
> www.dkfz.de <http://www.dkfz.de>
>
> Research for a Life without Cancer
>
> Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
> VAT-ID No.: DE143293537
>
>
>
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chipster-tech
>
>
>
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chipster-tech
-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

Re: [Chipster-tech] [Extern] - Re: Microarray PCA analysis not starting

[<chi...@cs...>]

Hi Oliver, 

Petri is out of office so just a quick note that yes, it might take some weeks until we could start a merge discussion. Is the phenodata line mainly in microarray analysis tools or have you replaced it in every tool? 

Best, 
Eija 


From: "Oliver Heil" <o....@dk...> 
To: "chipster-tech" <chi...@li...> 
Cc: "Konrad, Aline" <a.k...@dk...> 
Sent: Friday, 8 September, 2023 13:14:41 
Subject: Re: [Chipster-tech] [Extern] - Re: Microarray PCA analysis not starting 



Hi Petri, 



Or if you have your own git branch (check with 'git status'): 

cd ~/git/chipster-tools/ 
git switch master 
git pull 
git switch <your-own-branch> 
git merge master 
bash reload.bash 




This. 

I have my own branch 'dkfz-tools' here 
[ https://github.com/oheil/chipster-tools | https://github.com/oheil/chipster-tools ] 


I did as you said and it seems to be fine now. 

I am now in sync with master on master and ahead 3 commits on dkfz-tools 
Most changes in branch dkfz-tools I could make a PR, it's mainly this: 


BQ_BEGIN
phenodata <- read.table('phenodata.tsv', header=TRUE, sep='\t', quote='', as.is=TRUE, check.names=FALSE, comment.char='') 
BQ_END
instead of 


BQ_BEGIN
phenodata <- read.table("phenodata.tsv", header = TRUE, sep = "\t", as.is = TRUE) 
BQ_END


But there are also some changes, e.g. for special chip types, which you may not want. 

Do you want me to do a PR from dkfz-tools now? 
My guess is, that it wouldn't be merged right away but that we would go into 
a few discussions, which is not a problem on my side. But if you say, it may take weeks 
until we start a merge discussion I would say, ok, lets do it at some time in future... 

Regards, 

Oli 


Am 08.09.2023 um 07:02 schrieb Chipster: 

BQ_BEGIN
Hi Oli, 

interesting. It looks like you still have the old tool scripts somehow. Have you mounted the tool scripts from the host ( [ https://github.com/chipster/chipster-openshift/blob/master/k3s/tool-script-dev.md#mount-tool-scripts-from-the-host | https://github.com/chipster/chipster-openshift/blob/master/k3s/tool-script-dev.md#mount-tool-scripts-from-the-host ] ) , i.e. something like this in your values.yaml? 

tools: 
hostPath: /opt/chipster/toolbox/tools 

In that case you would have to also update the tool scripts on your host: 

cd ~/git/chipster-tools/ 
git pull 
bash reload.bash 

Unfortunately the reload script hasn't been very reliable. It should print some logs. If the log timestamps are not from the current UTC time, then the reload didn't happen. In that case, restart the whole toolbox and wait for the old pod to disappear: 

kubectl rollout restart deployment/toolbox 
watch kubectl get pod 

Or if you have your own git branch (check with 'git status'): 

cd ~/git/chipster-tools/ 
git switch master 
git pull 
git switch <your-own-branch> 
git merge master 
bash reload.bash 

If you haven't mounted the tools scripts from the host, then the tool scripts should come directly from the toolbox container image and the version tag should make sure you are running the latest version of it. I guess we should then check how to toolbox deployment looks like: 

kubectl get deployment/toolbox -o yaml 

Best regards, 
Petri 


BQ_END

-- 


Oliver Heil 
Microarray Core Facility 
Bioinformatics 


German Cancer Research Center (DKFZ) 
Foundation under Public Law 
Im Neuenheimer Feld 580 
69120 Heidelberg 
Germany 

[ mailto:o....@dk... | 
          o....@dk... ] 
[ https://www.dkfz.de/gpcf/support/ | 
            Support: www.dkfz.de/gpcf/support/ ] 
[ http://www.dkfz.de/ | www.dkfz.de ] 


Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich 
VAT-ID No.: DE143293537 


_______________________________________________ 
Chipster-tech mailing list 
Chi...@li... 
https://lists.sourceforge.net/lists/listinfo/chipster-tech 

Re: [Chipster-tech] [Extern] - Re: Microarray PCA analysis not starting

[Oliver H. <o....@dk...>]

Hi Petri,

> Or if you have your own git branch (check with 'git status'):
>
> cd ~/git/chipster-tools/
> git switch master
> git pull
> git switch <your-own-branch>
> git merge master
> bash reload.bash

This.

I have my own branch 'dkfz-tools' here
https://github.com/oheil/chipster-tools

I did as you said and it seems to be fine now.

I am now in sync with master on master and ahead 3 commits on dkfz-tools
Most changes in branch dkfz-tools I could make a PR, it's mainly this:

> phenodata <- read.table('phenodata.tsv', header=TRUE, sep='\t', 
> quote='', as.is=TRUE, check.names=FALSE, comment.char='')
instead of

> phenodata <- read.table("phenodata.tsv", header = TRUE, sep = "\t", 
> as.is = TRUE)

But there are also some changes, e.g. for special chip types, which you 
may not want.

Do you want me to do a PR from dkfz-tools now?
My guess is, that it wouldn't be merged right away but that we would go into
a few discussions, which is not a problem on my side. But if you say, it 
may take weeks
until we start a merge discussion I would say, ok, lets do it at some 
time in future...

Regards,

Oli


Am 08.09.2023 um 07:02 schrieb Chipster:
> Hi Oli,
>
> interesting. It looks like you still have the old tool scripts 
> somehow. Have you mounted the tool scripts from the host 
> (https://github.com/chipster/chipster-openshift/blob/master/k3s/tool-script-dev.md#mount-tool-scripts-from-the-host) 
> , i.e. something like this in your values.yaml?
>
>   tools:
>     hostPath: /opt/chipster/toolbox/tools
>
> In that case you would have to also update the tool scripts on your host:
>
> cd ~/git/chipster-tools/
> git pull
> bash reload.bash
>
> Unfortunately the reload script hasn't been very reliable. It should 
> print some logs. If the log timestamps are not from the current UTC 
> time, then the reload didn't happen. In that case, restart the whole 
> toolbox and wait for the old pod to disappear:
>
> kubectl rollout restart deployment/toolbox
> watch kubectl get pod
>
> Or if you have your own git branch (check with 'git status'):
>
> cd ~/git/chipster-tools/
> git switch master
> git pull
> git switch <your-own-branch>
> git merge master
> bash reload.bash
>
> If you haven't mounted the tools scripts from the host, then the tool 
> scripts should come directly from the toolbox container image and the 
> version tag should make sure you are running the latest version of it. 
> I guess we should then check how to toolbox deployment looks like:
>
> kubectl get deployment/toolbox -o yaml
>
> Best regards,
> Petri
>
-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

Re: [Chipster-tech] [Extern] - Re: Microarray PCA analysis not starting

[Chipster <chi...@cs...>]

Hi Oli,

interesting. It looks like you still have the old tool scripts somehow. Have you mounted the tool scripts from the host (https://github.com/chipster/chipster-openshift/blob/master/k3s/tool-script-dev.md#mount-tool-scripts-from-the-host) , i.e. something like this in your values.yaml?

  tools:
    hostPath: /opt/chipster/toolbox/tools

In that case you would have to also update the tool scripts on your host:

cd ~/git/chipster-tools/
git pull
bash reload.bash

Unfortunately the reload script hasn't been very reliable. It should print some logs. If the log timestamps are not from the current UTC time, then the reload didn't happen. In that case, restart the whole toolbox and wait for the old pod to disappear:

kubectl rollout restart deployment/toolbox
watch kubectl get pod

Or if you have your own git branch (check with 'git status'):

cd ~/git/chipster-tools/
git switch master
git pull
git switch <your-own-branch>
git merge master
bash reload.bash

If you haven't mounted the tools scripts from the host, then the tool scripts should come directly from the toolbox container image and the version tag should make sure you are running the latest version of it. I guess we should then check how to toolbox deployment looks like:

kubectl get deployment/toolbox -o yaml

Best regards,
Petri


> On 7. Sep 2023, at 16.06, Oliver Heil <o....@dk...> wrote:
> 
> Hi Petri,
> The update to 4.7.5 worked flawless, but:
> Still the docker image does not download, slightly different error:
> 
> ...
> QoS Class:                   Burstable
> Node-Selectors:              <none>
> Tolerations:                 node.kubernetes.io/not-ready:NoExecuteop=Exists for 300s
>                              node.kubernetes.io/unreachable:NoExecuteop=Exists for 300s
> Events:
>   Type     Reason     Age                From               Message
>   ----     ------     ----               ----               -------
>   Normal   Scheduled  95s                default-scheduler  Successfully assigned default/comp-job-d6b3f450-3cd1-459a-ac8a-6cd70cbaa009-ordination-pca.r to chipstervm1
>   Normal   BackOff    13s (x5 over 94s)  kubelet            Back-off pulling image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5"
>   Warning  Failed     13s (x5 over 94s)  kubelet            Error: ImagePullBackOff
>   Normal   Pulling    1s (x4 over 95s)   kubelet            Pulling image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5"
>   Warning  Failed     0s (x4 over 94s)   kubelet            Failed to pull image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5": rpc error: code = NotFound desc = failed to pull and unpack image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5": failed to resolve reference "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5":docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5: not found
>   Warning  Failed     0s (x4 over 94s)   kubelet            Error: ErrImagePull
> ...
> 
> 
> Am 07.09.2023 um 14:01 schrieb Chipster:
>> Hi Oli,
>> 
>> Good catch, thanks for spotting that!
>> 
>> In v4.7.2 we renamed all period characters "." in the image names to dashes "-", because an OpenShift command line tool didn't like the periods. We changed these also in the so called "runtime" configuration in https://github.com/chipster/chipster-web-server/blob/master/src/main/resources/chipster-defaults.yaml , but there were two tools which had an overriding image name also in the tool itself.  
>> 
>> This is now fixed in Chipster version v4.7.5.
>> 
>> Best regards,
>> Petri
>> 
> -- 
> Oliver Heil
> Microarray Core Facility
> Bioinformatics
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> o....@dk...
> Support: www.dkfz.de/gpcf/support/
> www.dkfz.deManagement Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
> VAT-ID No.: DE143293537
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chipster-tech

Re: [Chipster-tech] [Extern] - Re: Microarray PCA analysis not starting

[Oliver H. <o....@dk...>]

Hi Petri,

The update to 4.7.5 worked flawless, but:

Still the docker image does not download, slightly different error:


...
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
   Type     Reason     Age                From Message
   ----     ------     ----               ---- -------
   Normal   Scheduled  95s                default-scheduler Successfully 
assigned 
default/comp-job-d6b3f450-3cd1-459a-ac8a-6cd70cbaa009-ordination-pca.r 
to chipstervm1
   Normal   BackOff    13s (x5 over 94s)  kubelet Back-off pulling image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5"
   Warning  Failed     13s (x5 over 94s)  kubelet Error: ImagePullBackOff
   Normal   Pulling    1s (x4 over 95s)   kubelet Pulling image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5"
   Warning  Failed     0s (x4 over 94s)   kubelet Failed to pull image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5": 
rpc error: code = NotFound desc = failed to pull and unpack image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5": 
failed to resolve reference 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5": 
docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.5: 
not found
   Warning  Failed     0s (x4 over 94s)   kubelet Error: ErrImagePull
...



Am 07.09.2023 um 14:01 schrieb Chipster:
> Hi Oli,
>
> Good catch, thanks for spotting that!
>
> In v4.7.2 we renamed all period characters "." in the image names to 
> dashes "-", because an OpenShift command line tool didn't like the 
> periods. We changed these also in the so called "runtime" 
> configuration in 
> https://github.com/chipster/chipster-web-server/blob/master/src/main/resources/chipster-defaults.yaml , 
> but there were two tools which had an overriding image name also in 
> the tool itself.
>
> This is now fixed in Chipster version v4.7.5.
>
> Best regards,
> Petri
>
-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

Re: [Chipster-tech] Microarray PCA analysis not starting

[Chipster <chi...@cs...>]

Hi Oli,

Good catch, thanks for spotting that!

In v4.7.2 we renamed all period characters "." in the image names to dashes "-", because an OpenShift command line tool didn't like the periods. We changed these also in the so called "runtime" configuration in https://github.com/chipster/chipster-web-server/blob/master/src/main/resources/chipster-defaults.yaml , but there were two tools which had an overriding image name also in the tool itself.  

This is now fixed in Chipster version v4.7.5.

Best regards,
Petri

> On 7. Sep 2023, at 12.21, Oliver Heil <o....@dk...> wrote:
> 
> Hi Petri,
> 
> since the new version 4.7.2 the microarray PCA analysis is not working anymore.
> 
> This is what I get from the server after I start a PCA analysis:
> 
> 
> 
> ubuntu@chipstervm1:~$ kubectl get pod
> NAME                                                             READY   STATUS             RESTARTS        AGE
> ...
> comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r   0/1     ImagePullBackOff   0               5m14s
> 
> 
> 
> ubuntu@chipstervm1:~$ kubectl logs comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r
> Error from server (BadRequest): container "comp-job" in pod "comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r" is waiting to start: image can't be pulled
> 
> 
> ubuntu@chipstervm1:~$ kubectl describe pod comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r
> ...
> Events:
>   Type     Reason     Age                    From               Message
>   ----     ------     ----                   ----               -------
>   Normal   Scheduled  6m32s                  default-scheduler  Successfully assigned default/comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r to chipstervm1
>   Normal   Pulling    4m56s (x4 over 6m32s)  kubelet            Pulling image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2 <http://docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2>"
>   Warning  Failed     4m56s (x4 over 6m31s)  kubelet            Failed to pull image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2 <http://docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2>": rpc error: code = NotFound desc = failed to pull and unpack image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2 <http://docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2>": failed to resolve reference "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2 <http://docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2>": docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2: <http://docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2:> not found
>   Warning  Failed     4m56s (x4 over 6m31s)  kubelet            Error: ErrImagePull
>   Warning  Failed     4m43s (x6 over 6m30s)  kubelet            Error: ImagePullBackOff
>   Normal   BackOff    89s (x20 over 6m30s)   kubelet            Back-off pulling image "docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2 <http://docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2>"
> 
> 
> So it seems there is something missing at the docker image site?
> 
> Regards,
> 
> Oli
> 
> 
> 
> -- 
> Oliver Heil
> Microarray Core Facility
> Bioinformatics
> 
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> 
> o....@dk... <mailto:o....@dk...>
> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
> www.dkfz.de <http://www.dkfz.de/>
> Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
> VAT-ID No.: DE143293537
> 
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li... <mailto:Chi...@li...>
> https://lists.sourceforge.net/lists/listinfo/chipster-tech

[Chipster-tech] Microarray PCA analysis not starting

[Oliver H. <o....@dk...>]

Hi Petri,

since the new version 4.7.2 the microarray PCA analysis is not working 
anymore.

This is what I get from the server after I start a PCA analysis:


ubuntu@chipstervm1:~$ kubectl get pod
NAME READY   STATUS             RESTARTS        AGE
...
comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r 0/1     
ImagePullBackOff   0               5m14s


ubuntu@chipstervm1:~$ kubectl logs 
comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r
Error from server (BadRequest): container "comp-job" in pod 
"comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r" is 
waiting to start: image can't be pulled

ubuntu@chipstervm1:~$ kubectl describe pod 
comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r
...
Events:
   Type     Reason     Age                    From Message
   ----     ------     ----                   ---- -------
   Normal   Scheduled  6m32s                  default-scheduler 
Successfully assigned 
default/comp-job-f159fc1c-4f60-41cd-8622-dfa7af547b73-ordination-pca.r 
to chipstervm1
   Normal   Pulling    4m56s (x4 over 6m32s)  kubelet Pulling image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2"
   Warning  Failed     4m56s (x4 over 6m31s)  kubelet Failed to pull 
image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2": 
rpc error: code = NotFound desc = failed to pull and unpack image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2": 
failed to resolve reference 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2": 
docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2: 
not found
   Warning  Failed     4m56s (x4 over 6m31s)  kubelet Error: ErrImagePull
   Warning  Failed     4m43s (x6 over 6m30s)  kubelet Error: 
ImagePullBackOff
   Normal   BackOff    89s (x20 over 6m30s)   kubelet Back-off pulling 
image 
"docker-registry.rahti.csc.fi/chipster-images-release/comp-20.04-r-deps:v4.7.2"

So it seems there is something missing at the docker image site?

Regards,

Oli


-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

Re: [Chipster-tech] [Extern] - Chipster v4.7.2 and tagged container images

[Chipster <chi...@cs...>]

Hi Oli,

I'm happy to hear that this update didn't cause too much trouble. 

My original goal with the release names was that we should use release candidate (-rc) names like v4.7.3-rc1 to make sure everything works. I guess I wont write messages here on the email list about every minor update (usually every 2-4 weeks), but only when the update procedure changes, there is other migration work needed or there is some urgent update. So you should be safe to install any non-rc version, v4.7.3 in this case. 

Now in practice I have noticed that it's easy to break something also when making the final release. I don't know any perfect solution for this, but maybe you could check that the specific tag is at least 24 hours old in https://github.com/chipster/chipster-openshift/tags , which should give us enough time either to fix or remove the release, or notify you on the email list about the situation. Or if you have a spare server, you could update that first to see that the new version isn't completely broken. 

Thank you for enabling more people to use Chipster!

Best regards,
Petri

> On 31. Aug 2023, at 12.52, Oliver Heil <o....@dk...> wrote:
> 
> Hi Petri,
> 
> thanks for the answers and the fast improvements of the update doc.
> 
> Our second chipster server is now up to date too.
> 
> The same 3 pods like before remain after update, but are easily removed with 
> 
> kubectl delete pod ...
> 
> During update, when checking for newest release with
> 
> 
> curl -s https://docker-registry.rahti.csc.fi/v2/chipster-images-release/base/tags/list -H "Authorization: Bearer anonymous" | jq .tags[] -r
> 
> I get:
> 
> latest
> v4.7.2
> v4.7.3
> 
> But I stick with v4.7.2 as it isn't clear if one should already use v4.7.3 now. Just a minor irritation.
> 
> Summary: update successfully finished :-)
> Thanks to all for all your hard and good work!
> 
> Best Regards,
> 
> Oli
> 
> 
> 
> Am 28.08.2023 um 16:57 schrieb Chipster:
>> Hi Oli,
>> 
>> answers below:
>> 
>>> On 25. Aug 2023, at 14.46, Oliver Heil <o....@dk...> <mailto:o....@dk...> wrote:
>>> 1)
>>> 
>>>> image:
>>>>   tag: v4.7.2
>>> to ~/values.yaml
>>> Should this be removed after successful update again? Or should it stay there?
>> It should stay there. It configures the containers to use images like "docker-registry.rahti.csc.fi/chipster-images-release/web-server:v4.7.2 <http://docker-registry.rahti.csc.fi/chipster-images-release/web-server:v4.7.2>", which makes 
>> sure that only this specific version is used until you decide to change it.
>> 
>>> 2)
>>> There seem to be 3 remnants from the old version (top 3 of below list):
>>> 
>>>> Every 2.0s: kubectl get pod
>>>> 
>>>> NAME                                READY   STATUS                   RESTARTS       AGE
>>>> type-service-7ff4944dd6-zzdrl       0/1     ContainerStatusUnknown   7 (33m ago)    21d
>>>> toolbox-86f7994cf9-wl6lc            0/1     Error                    12 (33m ago)   129d
>>>> backup-77c55fc4df-7ccl4             0/1     Error                    11             129d
>>> Is there something left to do with them?
>> I don't know why K3s haven't removed those. You could try remove them yourself:
>> 
>> kubectl delete pod type-service-7ff4944dd6-zzdrl
>> 
>> And so on. However, I don't think those containers really exist anymore, so those shouldn't cause any harm apart from making your listing uglier.
>> 
>>> Some further suggestions:
>>> 1) You linked to the #updates tag in the k3s/README.md document. For the new images one needs to jump
>>> ahead/up in the same document following the link in "If there is a newer Chipster container image version available",
>>> which is somehow confusing. My suggestion is: there should be a single "UPDATE" document which can be
>>> followed from top to down. The only jumps in there should be when skipping something not necessary and the
>>> jump should never be ahead/up but always downwards.
>> I tried to make it easy for the first time installers to specify the container image version, but I admit that end result wasn't very clear for the updates. I have now merged the two chapters.
>> 
>>> 2) It isn't clear in which folder the update procedure has to be started. Where should I be, when I do the first
>>> step "git pull" ? Actually it is "~/git/chipster-openshift/k3s", I always have to explore it again, because last time
>>> I had to is months ago.
>> Added there. It's definitely a good place to mention it, because you hopefully are able to update the Chipster server much more often than you have to reinstall it (where it's only mentioned now).
>> 
>>> 3) Add something about updating the Ubuntu system to the Update-Document. Currently we run 
>>> Ubuntu 20.04.6 LTS on the chipster server. A 'do-release-upgrade' is shown at login, of course
>>> I hesitate to do this. Some information would be nice, if chipster is still considerung Ubuntu 20.04 as
>>> best operating system or if it is already safe to upgrade to a newer version. 
>> Ubuntu releases long term support (LTS) versions every two years, e.g. 20.04, 22.04 and 24.04. Each version receives updates for five years. Our plan is to minimize the migration work by skipping every other LTS release. So we are using 20.04 for now and plan to migrate to 24.04 after it's released in April 2024 and before the 20.04 end of life in April 2025. I added a few words to the update instructions about this.
>> 
>>> 4) Can I see any version information in the chipster webgui? Like, if a user wants to see the
>>> current version(s) of the system? I did a brief search for it and didn't found anything. It isn't
>>> that much important but somehow some kind of software standard, to expose version information
>>> to the user. I think there are several versions (chipster, chipster-tools,...) to be shown somewhere.
>> You are right, the version information isn't really available for the end-users at the moment like it should be. We would have to build some kind of automatic system to take the version information from git and incorporate it in the container images. Then the web app could show its own version and perhaps query to backend versions from the Rest APIs. I think the latest change in container image versioning forces us to be more stringent about release versioning, which could be a good stepping stone into that direction. I wrote this down on our backlog and let's hope we can improve it at some point.
>> 
>> Thanks for the comments! Let's hope the instructions are now a bit easier to follow.
>> 
>> Best regards,
>> Petri
>> 
> -- 
> Oliver Heil
> Microarray Core Facility
> Bioinformatics
> 
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> 
> o....@dk... <mailto:o....@dk...>
> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
> www.dkfz.de <http://www.dkfz.de/>
> Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
> VAT-ID No.: DE143293537
> 
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li... <mailto:Chi...@li...>
> https://lists.sourceforge.net/lists/listinfo/chipster-tech

Re: [Chipster-tech] [Extern] - Chipster v4.7.2 and tagged container images

[Oliver H. <o....@dk...>]

Hi Petri,

thanks for the answers and the fast improvements of the update doc.

Our second chipster server is now up to date too.

The same 3 pods like before remain after update, but are easily removed 
with

kubectl delete pod ...

During update, when checking for newest release with

curl -s 
https://docker-registry.rahti.csc.fi/v2/chipster-images-release/base/tags/list 
-H "Authorization: Bearer anonymous" | jq .tags[] -r

I get:

latest
v4.7.2
v4.7.3

But I stick with v4.7.2 as it isn't clear if one should already use 
v4.7.3 now. Just a minor irritation.

Summary: update successfully finished :-)
Thanks to all for all your hard and good work!

Best Regards,

Oli


Am 28.08.2023 um 16:57 schrieb Chipster:
> Hi Oli,
>
> answers below:
>
>> On 25. Aug 2023, at 14.46, Oliver Heil<o....@dk...>  wrote:
>> 1)
>>
>>> image:
>>>    tag: v4.7.2
>> to ~/values.yaml
>> Should this be removed after successful update again? Or should it stay there?
>
> It should stay there. It configures the containers to use images like "docker-registry.rahti.csc.fi/chipster-images-release/web-server:v4.7.2", which makes
> sure that only this specific version is used until you decide to change it.
>
>> 2)
>> There seem to be 3 remnants from the old version (top 3 of below list):
>>
>>> Every 2.0s: kubectl get pod
>>>
>>> NAME                                READY   STATUS                   RESTARTS       AGE
>>> type-service-7ff4944dd6-zzdrl       0/1     ContainerStatusUnknown   7 (33m ago)    21d
>>> toolbox-86f7994cf9-wl6lc            0/1     Error                    12 (33m ago)   129d
>>> backup-77c55fc4df-7ccl4             0/1     Error                    11             129d
>>
>> Is there something left to do with them?
> I don't know why K3s haven't removed those. You could try remove them yourself:
>
> kubectl delete pod type-service-7ff4944dd6-zzdrl
>
> And so on. However, I don't think those containers really exist anymore, so those shouldn't cause any harm apart from making your listing uglier.
>
>> Some further suggestions:
>> 1) You linked to the #updates tag in the k3s/README.md document. For the new images one needs to jump
>> ahead/up in the same document following the link in "If there is a newer Chipster container image version available",
>> which is somehow confusing. My suggestion is: there should be a single "UPDATE" document which can be
>> followed from top to down. The only jumps in there should be when skipping something not necessary and the
>> jump should never be ahead/up but always downwards.
> I tried to make it easy for the first time installers to specify the container image version, but I admit that end result wasn't very clear for the updates. I have now merged the two chapters.
>
>> 2) It isn't clear in which folder the update procedure has to be started. Where should I be, when I do the first
>> step "git pull" ? Actually it is "~/git/chipster-openshift/k3s", I always have to explore it again, because last time
>> I had to is months ago.
> Added there. It's definitely a good place to mention it, because you hopefully are able to update the Chipster server much more often than you have to reinstall it (where it's only mentioned now).
>
>> 3) Add something about updating the Ubuntu system to the Update-Document. Currently we run
>> Ubuntu 20.04.6 LTS on the chipster server. A 'do-release-upgrade' is shown at login, of course
>> I hesitate to do this. Some information would be nice, if chipster is still considerung Ubuntu 20.04 as
>> best operating system or if it is already safe to upgrade to a newer version.
> Ubuntu releases long term support (LTS) versions every two years, e.g. 20.04, 22.04 and 24.04. Each version receives updates for five years. Our plan is to minimize the migration work by skipping every other LTS release. So we are using 20.04 for now and plan to migrate to 24.04 after it's released in April 2024 and before the 20.04 end of life in April 2025. I added a few words to the update instructions about this.
>
>> 4) Can I see any version information in the chipster webgui? Like, if a user wants to see the
>> current version(s) of the system? I did a brief search for it and didn't found anything. It isn't
>> that much important but somehow some kind of software standard, to expose version information
>> to the user. I think there are several versions (chipster, chipster-tools,...) to be shown somewhere.
> You are right, the version information isn't really available for the end-users at the moment like it should be. We would have to build some kind of automatic system to take the version information from git and incorporate it in the container images. Then the web app could show its own version and perhaps query to backend versions from the Rest APIs. I think the latest change in container image versioning forces us to be more stringent about release versioning, which could be a good stepping stone into that direction. I wrote this down on our backlog and let's hope we can improve it at some point.
>
> Thanks for the comments! Let's hope the instructions are now a bit easier to follow.
>
> Best regards,
> Petri
>
-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

Re: [Chipster-tech] [Extern] - Chipster v4.7.2 and tagged container images

[Chipster <chi...@cs...>]

Hi Oli,

answers below:

> On 25. Aug 2023, at 14.46, Oliver Heil <o....@dk...> wrote:
> 1)
> 
>> image:
>>   tag: v4.7.2
> 
> to ~/values.yaml
> Should this be removed after successful update again? Or should it stay there?


It should stay there. It configures the containers to use images like "docker-registry.rahti.csc.fi/chipster-images-release/web-server:v4.7.2", which makes 
sure that only this specific version is used until you decide to change it.

> 2)
> There seem to be 3 remnants from the old version (top 3 of below list):
> 
>> Every 2.0s: kubectl get pod
>> 
>> NAME                                READY   STATUS                   RESTARTS       AGE
>> type-service-7ff4944dd6-zzdrl       0/1     ContainerStatusUnknown   7 (33m ago)    21d
>> toolbox-86f7994cf9-wl6lc            0/1     Error                    12 (33m ago)   129d
>> backup-77c55fc4df-7ccl4             0/1     Error                    11             129d
> 
> 
> Is there something left to do with them?

I don't know why K3s haven't removed those. You could try remove them yourself:

kubectl delete pod type-service-7ff4944dd6-zzdrl

And so on. However, I don't think those containers really exist anymore, so those shouldn't cause any harm apart from making your listing uglier.

> Some further suggestions:
> 1) You linked to the #updates tag in the k3s/README.md document. For the new images one needs to jump
> ahead/up in the same document following the link in "If there is a newer Chipster container image version available",
> which is somehow confusing. My suggestion is: there should be a single "UPDATE" document which can be
> followed from top to down. The only jumps in there should be when skipping something not necessary and the
> jump should never be ahead/up but always downwards.

I tried to make it easy for the first time installers to specify the container image version, but I admit that end result wasn't very clear for the updates. I have now merged the two chapters.

> 2) It isn't clear in which folder the update procedure has to be started. Where should I be, when I do the first
> step "git pull" ? Actually it is "~/git/chipster-openshift/k3s", I always have to explore it again, because last time
> I had to is months ago.

Added there. It's definitely a good place to mention it, because you hopefully are able to update the Chipster server much more often than you have to reinstall it (where it's only mentioned now).

> 3) Add something about updating the Ubuntu system to the Update-Document. Currently we run 
> Ubuntu 20.04.6 LTS on the chipster server. A 'do-release-upgrade' is shown at login, of course
> I hesitate to do this. Some information would be nice, if chipster is still considerung Ubuntu 20.04 as
> best operating system or if it is already safe to upgrade to a newer version. 

Ubuntu releases long term support (LTS) versions every two years, e.g. 20.04, 22.04 and 24.04. Each version receives updates for five years. Our plan is to minimize the migration work by skipping every other LTS release. So we are using 20.04 for now and plan to migrate to 24.04 after it's released in April 2024 and before the 20.04 end of life in April 2025. I added a few words to the update instructions about this.

> 4) Can I see any version information in the chipster webgui? Like, if a user wants to see the
> current version(s) of the system? I did a brief search for it and didn't found anything. It isn't
> that much important but somehow some kind of software standard, to expose version information
> to the user. I think there are several versions (chipster, chipster-tools,...) to be shown somewhere.

You are right, the version information isn't really available for the end-users at the moment like it should be. We would have to build some kind of automatic system to take the version information from git and incorporate it in the container images. Then the web app could show its own version and perhaps query to backend versions from the Rest APIs. I think the latest change in container image versioning forces us to be more stringent about release versioning, which could be a good stepping stone into that direction. I wrote this down on our backlog and let's hope we can improve it at some point.

Thanks for the comments! Let's hope the instructions are now a bit easier to follow.

Best regards,
Petri

Re: [Chipster-tech] [Extern] - Chipster v4.7.2 and tagged container images

[Oliver H. <o....@dk...>]

Hi Petri,

I followed the update instruction, chipster seems to run, but two 
questions remain:

1)

I added

> image:
>   tag: v4.7.2

to ~/values.yaml

Should this be removed after successful update again? Or should it stay 
there?

2)

There seem to be 3 remnants from the old version (top 3 of below list):

> Every 2.0s: kubectl get pod
>
> NAME                                READY STATUS                   
> RESTARTS       AGE
> type-service-7ff4944dd6-zzdrl       0/1 ContainerStatusUnknown   7 
> (33m ago)    21d
> toolbox-86f7994cf9-wl6lc            0/1 Error                    12 
> (33m ago)   129d
> backup-77c55fc4df-7ccl4             0/1 Error                    
> 11             129d
> chipster-session-db-postgresql-0    1/1 Running                  
> 0              27m
> chipster-job-history-postgresql-0   1/1 Running                  
> 0              27m
> chipster-auth-postgresql-0          1/1 Running                  
> 0              26m
> auth-9d654f456-9h9x8                1/1 Running                  3 
> (22m ago)    26m
> service-locator-57c4fd5486-5gptt    1/1 Running                  4 
> (21m ago)    26m
> session-db-66996d7bf9-64l7m         1/1 Running                  4 
> (21m ago)    26m
> toolbox-c88ccc85d-fr6wx             1/1 Running                  4 
> (21m ago)    26m
> backup-dc7fdc458-v2vxh              1/1 Running                  4 
> (21m ago)    26m
> file-storage-0                      1/1 Running                  4 
> (21m ago)    22m
> type-service-6c6db85655-bb69q       1/1 Running                  4 
> (21m ago)    26m
> scheduler-66fb56bcd6-xkpm4          1/1 Running                  5 
> (21m ago)    26m
> session-worker-6f49c987b6-5z6hp     1/1 Running                  5 
> (20m ago)    26m
> job-history-689477cc4c-5tqmp        1/1 Running                  5 
> (21m ago)    26m
> web-server-84f8784bc9-ksjjx         1/1 Running                  5 
> (20m ago)    26m
> file-broker-7679bf556b-vp2vb        1/1 Running                  5 
> (20m ago)    26m

Is there something left to do with them?

Some further suggestions:

1) You linked to the #updates tag in the k3s/README.md document. For the 
new images one needs to jump
ahead/up in the same document following the link in "If there is a newer 
Chipster container image version available",
which is somehow confusing. My suggestion is: there should be a single 
"UPDATE" document which can be
followed from top to down. The only jumps in there should be when 
skipping something not necessary and the
jump should never be ahead/up but always downwards.

2) It isn't clear in which folder the update procedure has to be 
started. Where should I be, when I do the first
step "git pull" ? Actually it is "~/git/chipster-openshift/k3s", I 
always have to explore it again, because last time
I had to is months ago.

3) Add something about updating the Ubuntu system to the 
Update-Document. Currently we run
Ubuntu 20.04.6 LTS on the chipster server. A 'do-release-upgrade' is 
shown at login, of course
I hesitate to do this. Some information would be nice, if chipster is 
still considerung Ubuntu 20.04 as
best operating system or if it is already safe to upgrade to a newer 
version.

4) Can I see any version information in the chipster webgui? Like, if a 
user wants to see the
current version(s) of the system? I did a brief search for it and didn't 
found anything. It isn't
that much important but somehow some kind of software standard, to 
expose version information
to the user. I think there are several versions (chipster, 
chipster-tools,...) to be shown somewhere.

Best regards,

Oli

Am 24.08.2023 um 10:35 schrieb Chipster:
> Hi,
>
> Chipster v4.7.2 has been released. This version includes a small, but incompatible change in the Chipster Rest API, so this requires you to update the backend services and the web app at the same time.
>
> We have started to add version tags to our container images to avoid inadvertant changes in your Chipster. Please follow the update instructions inhttps://github.com/chipster/chipster-openshift/blob/k3s/k3s/README.md#updates  to start using these new tagged images in your installation. The instructions now also ask you to pull all images at once, which additionally makes sure that your installation won't depend on our image registry anymore after the installation.
>
> The container image repository has changed, but the update instructions will take care of this for you. The old Helm templates used the latest images in a image repository docker-registry.rahti.csc.fi/chipster-images/ . We didn't want to push the new images in the same repository, because it could have broken the current installations. From now own the images are released in a new repository docker-registry.rahti.csc.fi/chipster-images-release/ . We'll try to keep the old repository available until end of year 2023. Let us know if you have problems in the update procedure so that we can try to sort them out before this.
>
> When testing the update procedure, my ingress refused any new connections until Ubuntu was rebooted. I don't know why this happened because we didn't change the K3s version or ingress configuration. Just don't be alarmed if this happens until you have rebooted Ubuntu.
>
> Technical details about the API change:
>
> Chipster has a few API endpoints, where the UserID is sent in the URL. For example the web app used to make a request to get the details of the User object from a URL /auth/users/jaas%2F<username> . Unfortunately the UserID (jaas%2F<username>) contains a forward slash character "/", which is also a separator in the URL path. Until now we tried to URL encode the slash to "%2F". This worked fine in the Kubernetes Ingress Controller Traefik, used in K3s. However, this prevented Chipster deployments in other Kubernetes releases, where NGINX Ingress Controller is used, because the NGINX (understandably) decoded the URL path already before the Chipster endpoint. We have now changed these API endpoints to pass the UserID in the query parameter instead: /auth/users?userId=jaas%2F<username> . Albeit breaking Rest API conventions, be hope this change makes Chipster easier to deploy in different environments.
>
> Best regards,
> Petri / Chipster team
>
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chipster-tech
-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

[Chipster-tech] Chipster v4.7.2 and tagged container images

[Chipster <chi...@cs...>]

Hi,

Chipster v4.7.2 has been released. This version includes a small, but incompatible change in the Chipster Rest API, so this requires you to update the backend services and the web app at the same time.

We have started to add version tags to our container images to avoid inadvertant changes in your Chipster. Please follow the update instructions in https://github.com/chipster/chipster-openshift/blob/k3s/k3s/README.md#updates to start using these new tagged images in your installation. The instructions now also ask you to pull all images at once, which additionally makes sure that your installation won't depend on our image registry anymore after the installation.

The container image repository has changed, but the update instructions will take care of this for you. The old Helm templates used the latest images in a image repository docker-registry.rahti.csc.fi/chipster-images/ . We didn't want to push the new images in the same repository, because it could have broken the current installations. From now own the images are released in a new repository docker-registry.rahti.csc.fi/chipster-images-release/ . We'll try to keep the old repository available until end of year 2023. Let us know if you have problems in the update procedure so that we can try to sort them out before this.

When testing the update procedure, my ingress refused any new connections until Ubuntu was rebooted. I don't know why this happened because we didn't change the K3s version or ingress configuration. Just don't be alarmed if this happens until you have rebooted Ubuntu.

Technical details about the API change: 

Chipster has a few API endpoints, where the UserID is sent in the URL. For example the web app used to make a request to get the details of the User object from a URL /auth/users/jaas%2F<username> . Unfortunately the UserID (jaas%2F<username>) contains a forward slash character "/", which is also a separator in the URL path. Until now we tried to URL encode the slash to "%2F". This worked fine in the Kubernetes Ingress Controller Traefik, used in K3s. However, this prevented Chipster deployments in other Kubernetes releases, where NGINX Ingress Controller is used, because the NGINX (understandably) decoded the URL path already before the Chipster endpoint. We have now changed these API endpoints to pass the UserID in the query parameter instead: /auth/users?userId=jaas%2F<username> . Albeit breaking Rest API conventions, be hope this change makes Chipster easier to deploy in different environments. 

Best regards,
Petri / Chipster team

Re: [Chipster-tech] [Extern] - Type-service starup problem solved

[Oliver H. <o....@dk...>]

Awesome, Petri!

Am 03.08.2023 um 16:29 schrieb Chipster:
> Hi all,
>
> for a long time we have had a strange problem where the type-service doesn't start, if more than a week has passed since the last image build. I believe I have finally sorted it out.
>
>
-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

[Chipster-tech] Type-service starup problem solved

[Chipster <chi...@cs...>]

Hi all,

for a long time we have had a strange problem where the type-service doesn't start, if more than a week has passed since the last image build. I believe I have finally sorted it out. 

I have now updated the images in the image repository and you shouldn't see this problem anymore after you pull the latest image one more time. You can follow the normal Chipster update procedure, or just pull this one image with the following commands. You can do it straight away or when your type-service refuses to start next time.

sudo k3s crictl pull docker-registry.rahti.csc.fi/chipster-images/chipster-web-server-js
kubectl rollout restart deployment/type-service

The issue seemed to be that the NodeJS package installation creates a folder ~/.npm containing some package caches and a timestamp of a last update check. In our case the package installation is defined in Dockerfile and runs as a root user, which ends up owning this .npm directory too. Later the type-service is run as a different user. Apparently NodeJS checks the previous timestamp when it starts and tries to check updates if a week or so has passed since the previous check. When that happened, it failed to update the .npm directory because of the wrong file owner. I have now both disabled the update checks and deleted the .npm folder.

I was able to reproduce the original problem and validate this fix by changing the virtual machine clock to a future date. Let's hope this fix still works when a week has passed for real.

Best regards,
Petri

Re: [Chipster-tech] no ingress in fresh chipster that des not work

[Chipster <chi...@cs...>]

Hi Pavel,

sorry for the delay. I don't know straight away what could cause those symptoms, but here are my first guesses and comments.

Could the server have reached one year without K3s restart? The K3s has some internal certificates which are valid for one year, but it should update those when it's restarted (more in https://docs.k3s.io/advanced).
You can try to restart it:

  sudo systemctl restart k3

If this doesn't help already, maybe you should delete all K3s pods and wait until new pods are running:

  kubectl delete pod --all -n kube-system
  watch kubectl get pod -n kube-system

And maybe the same for all Chipster pods too:

  kubectl delete pod --all
  watch kubectl get pod

Alternatively, rebooting the whole Ubuntu would make sure that everything is restarted. 

If the kubectl command stops working at some point, you probably have to get a new token for it (https://sourceforge.net/p/chipster/mailman/message/37692383/):

   sudo kubectl config view --raw > ~/.kube/config


If this wasn't caused by expired K3s certificates, then we have to dig deeper. Having containers evicted or in ContainerStatusUknown state is certainly unexpected. Maybe K3s or containerd logs could shed light on this:

  journalctl -u k3s | tail
  sudo tail /var/lib/rancher/k3s/agent/containerd/containerd.log

You could check also if the pod events show any hints:

  kubectl describe pod POD_NAME

How much RAM memory does the server have? Could the lack of memory cause those containers to fail?


The Ingress really would be the first thing to check when you don't get http responses even if containers are running. A few years ago the Traefik in K3s started to use its own IngressRoute objects instead of the standard Kubernetes Ingress objects. It looks like we still had the old documentation in https://github.com/chipster/chipster-openshift/blob/k3s/k3s/getting-started-with-k3s.md#debugging-commands, but I have now updated it.


There were some weird K3s update problems in the spring (https://sourceforge.net/p/chipster/mailman/chipster-tech/?viewmonth=202304). Let's check your K3s version to rule out those:

$ k3s -version
k3s version v1.26.4+k3s1 (8d0255af)
go version go1.19.8

Best regards,
Petri

> On 14. Jul 2023, at 18.03, Pavel Fibich via Chipster-tech <chi...@li...> wrote:
> 
> Dear Chipster developers,
> 
> after some time Chipster standing on Debian stop working, I tried to repair it, but finally I end up with fresh installation after removing of ansible, kubectl etc.  that is also not working. I started with git (I did pull today) and then (no errors there)
> 
> ansible-playbook ansible/install-deps.yml -i "localhost," -c local -e user=$(whoami)
> bash generate-passwords.bash
> helm dependency update helm/chipster
> bash deploy.bash -f ~/values.yaml
> 
> in values.yaml, there is only hostname, toolsBin, and auth settings for oidc. By watching pods, there are all types of pods running (some of them are multiple times eg. type-service, with the other version in evicted or ContainerStatusUnknown state, but all the time there is one of each type running). But chipster is not accessible via http page (web-server pod is running). Surprisingly, there are no ingress, I observed that by kubectl get ingress : "No resources found in default namespace." Do you have any idea where the problem could be?
> 
> I tried to remove everything according your Uninstall guide, also with purging ansible from the system. But without success.
> 
> Best, Pavel
> 
> 
> 
> 
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chipster-tech

Re: [Chipster-tech] no ingress in fresh chipster that des not work

[<chi...@cs...>]

Hi Pavel,

Thank you for reporting the problem. Unfortunately our technical guys are on holidays now, but they will look into this as soon as they return on 31.7.

Best,
Eija

----- Original Message -----
From: "Pavel Fibich via Chipster-tech" <chi...@li...>
To: "chipster-tech" <chi...@li...>
Cc: "pavel fibich" <pav...@ce...>
Sent: Friday, 14 July, 2023 18:03:19
Subject: [Chipster-tech] no ingress in fresh chipster that des not work

Dear Chipster developers,

after some time Chipster standing on Debian stop working, I tried to 
repair it, but finally I end up with fresh installation after removing 
of ansible, kubectl etc.  that is also not working. I started with git 
(I did pull today) and then (no errors there)

ansible-playbook ansible/install-deps.yml -i "localhost," -c local -e 
user=$(whoami)
bash generate-passwords.bash
helm dependency update helm/chipster
bash deploy.bash -f ~/values.yaml

in values.yaml, there is only hostname, toolsBin, and auth settings for 
oidc. By watching pods, there are all types of pods running (some of 
them are multiple times eg. type-service, with the other version in 
evicted or ContainerStatusUnknown state, but all the time there is one 
of each type running). But chipster is not accessible via http page 
(web-server pod is running). Surprisingly, there are no ingress, I 
observed that by kubectl get ingress : "No resources found in default 
namespace." Do you have any idea where the problem could be?

I tried to remove everything according your Uninstall guide, also with 
purging ansible from the system. But without success.

Best, Pavel




_______________________________________________
Chipster-tech mailing list
Chi...@li...
https://lists.sourceforge.net/lists/listinfo/chipster-tech

[Chipster-tech] no ingress in fresh chipster that des not work

[Pavel F. <pav...@ce...>]

Dear Chipster developers,

after some time Chipster standing on Debian stop working, I tried to 
repair it, but finally I end up with fresh installation after removing 
of ansible, kubectl etc.  that is also not working. I started with git 
(I did pull today) and then (no errors there)

ansible-playbook ansible/install-deps.yml -i "localhost," -c local -e 
user=$(whoami)
bash generate-passwords.bash
helm dependency update helm/chipster
bash deploy.bash -f ~/values.yaml

in values.yaml, there is only hostname, toolsBin, and auth settings for 
oidc. By watching pods, there are all types of pods running (some of 
them are multiple times eg. type-service, with the other version in 
evicted or ContainerStatusUnknown state, but all the time there is one 
of each type running). But chipster is not accessible via http page 
(web-server pod is running). Surprisingly, there are no ingress, I 
observed that by kubectl get ingress : "No resources found in default 
namespace." Do you have any idea where the problem could be?

I tried to remove everything according your Uninstall guide, also with 
purging ansible from the system. But without success.

Best, Pavel

Re: [Chipster-tech] Chipstet upload analyzed data file

[Oliver H. <o....@dk...>]

Dear Qian,

Have you tried "Import session file" on the "Sessions" Tab/Page?

If yes what error comes up?

Regards,

Oli

Am 28.06.2023 um 13:43 schrieb Sun, Qian:
>
> Dear Sir, dear Madam,
>
> I did some microarray data analysis via chipster software in 2018, and 
> I exported the whole analysis as one zip file.
>
> And now I would like reanalysis it but I don’t know how to import the 
> analyzed data into the online web app. I wonder if you could help me 
> with that?
>
> Thank you very much and best regards,
>
> Qian Sun (DKFZ, Heidelberg, Germany)
>
>
>
> _______________________________________________
> Chipster-tech mailing list
> Chi...@li...
> https://lists.sourceforge.net/lists/listinfo/chipster-tech
-- 

Oliver Heil
Microarray Core Facility
Bioinformatics

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany

o....@dk... <mailto:o....@dk...>
Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/>
www.dkfz.de <http://www.dkfz.de>

Research for a Life without Cancer

Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich
VAT-ID No.: DE143293537

[Chipster-tech] Chipstet upload analyzed data file

[Sun, Q. <q....@dk...>]

Dear Sir, dear Madam,
I did some microarray data analysis via chipster software in 2018, and I exported the whole analysis as one zip file.
And now I would like reanalysis it but I don't know how to import the analyzed data into the online web app. I wonder if you could help me with that?
Thank you very much and best regards,

Qian Sun (DKFZ, Heidelberg, Germany)

[Chipster-tech] cert-manager updated

[Chipster <chi...@cs...>]

Hi all,

Chipster's instructions for TLS (https) configuration used to install an old version v1.3.1 of cert-manager, which didn't work anymore. 

I have updated the scripts and instructions to install the latest v1.11.0. cert-manager. If you are using TLS in Chipster, please run "git pull" to get the latest version of the deployment scripts and then follow the instruction page https://github.com/chipster/chipster-openshift/blob/k3s/k3s/tls.md to install the new cert-manager. If you haven't configured TLS, no action is needed. 

I tested this update procedure both with and without the TLS configuration. However, I wasn't able to test what happens if you already have an old certificate present, because the old version doesn't issue certificates anymore. 

Like always, let us know if you find any problems. 

Best regards,
Petri