You can subscribe to this list here.
2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
(10) |
Nov
|
Dec
(2) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2009 |
Jan
(4) |
Feb
(1) |
Mar
(6) |
Apr
(1) |
May
(1) |
Jun
|
Jul
(1) |
Aug
(2) |
Sep
(1) |
Oct
|
Nov
(1) |
Dec
(1) |
2010 |
Jan
(3) |
Feb
|
Mar
|
Apr
(2) |
May
(4) |
Jun
(8) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(2) |
Dec
(5) |
2011 |
Jan
(1) |
Feb
(1) |
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(7) |
Nov
(6) |
Dec
|
2012 |
Jan
|
Feb
|
Mar
|
Apr
(2) |
May
|
Jun
(15) |
Jul
(8) |
Aug
(10) |
Sep
(11) |
Oct
(4) |
Nov
(3) |
Dec
(3) |
2013 |
Jan
(7) |
Feb
(11) |
Mar
(3) |
Apr
(3) |
May
|
Jun
(2) |
Jul
|
Aug
(1) |
Sep
(3) |
Oct
|
Nov
(2) |
Dec
(5) |
2014 |
Jan
(5) |
Feb
(6) |
Mar
(1) |
Apr
|
May
(1) |
Jun
(3) |
Jul
|
Aug
(1) |
Sep
|
Oct
(7) |
Nov
(10) |
Dec
(14) |
2015 |
Jan
(14) |
Feb
(6) |
Mar
(22) |
Apr
(11) |
May
(9) |
Jun
(4) |
Jul
(6) |
Aug
|
Sep
(2) |
Oct
|
Nov
(2) |
Dec
(4) |
2016 |
Jan
(14) |
Feb
|
Mar
(15) |
Apr
(3) |
May
(2) |
Jun
(1) |
Jul
(5) |
Aug
(2) |
Sep
(7) |
Oct
(7) |
Nov
(4) |
Dec
(5) |
2017 |
Jan
(6) |
Feb
(6) |
Mar
(2) |
Apr
|
May
(8) |
Jun
(15) |
Jul
(4) |
Aug
|
Sep
(4) |
Oct
|
Nov
(6) |
Dec
|
2018 |
Jan
(13) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(12) |
2019 |
Jan
(1) |
Feb
(2) |
Mar
|
Apr
(3) |
May
(4) |
Jun
|
Jul
(4) |
Aug
(2) |
Sep
(3) |
Oct
(4) |
Nov
(2) |
Dec
|
2020 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(7) |
Jul
|
Aug
|
Sep
|
Oct
(7) |
Nov
(1) |
Dec
(2) |
2021 |
Jan
(2) |
Feb
|
Mar
(2) |
Apr
(2) |
May
|
Jun
(1) |
Jul
(9) |
Aug
(4) |
Sep
|
Oct
|
Nov
|
Dec
(2) |
2022 |
Jan
|
Feb
|
Mar
(10) |
Apr
(5) |
May
(10) |
Jun
(9) |
Jul
(10) |
Aug
(7) |
Sep
|
Oct
|
Nov
(12) |
Dec
(20) |
2023 |
Jan
(1) |
Feb
|
Mar
|
Apr
(13) |
May
(3) |
Jun
(2) |
Jul
(2) |
Aug
(8) |
Sep
(7) |
Oct
|
Nov
|
Dec
|
2024 |
Jan
(1) |
Feb
(2) |
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(2) |
Sep
(20) |
Oct
(4) |
Nov
|
Dec
|
From: Pavlin M. <pav...@up...> - 2023-05-11 12:56:27
|
Dear support, I have just attempted to install CHIPSTER on an OpenStack 16CPU/16GBRAM instance running Ubuntu 20.04 following the instruction on https://github.com/chipster/chipster-openshift/tree/master/k3s The setup appears working, but trying to configure with bash deploy.bash -f ~/values.yaml --set deployments.comp.configs.comp-max-jobs=\"3\" reveals a problem Error: UPGRADE FAILED: execution error at (chipster/templates/auth-secret.yaml:18:49): deployments.*.password is required below are some details that could help describing the situation. ubuntu@chipster:~/git/chipster-openshift/k3s$ kubectl get pod NAME READY STATUS RESTARTS AGE download-tools-bin-chipster-4-5-2-bvjb8 0/1 Completed 0 17h chipster-auth-postgresql-0 1/1 Running 2 (166m ago) 22h chipster-session-db-postgresql-0 1/1 Running 2 (166m ago) 22h chipster-job-history-postgresql-0 1/1 Running 2 (166m ago) 22h auth-68744cdc54-qq8l4 1/1 Running 0 150m service-locator-d747c6ff9-gq6pt 1/1 Running 0 150m session-db-7cdf7cbc5b-jjfrq 1/1 Running 0 150m session-worker-66464d4746-qqq56 1/1 Running 0 150m web-server-69c489478d-sx4z4 1/1 Running 0 150m backup-8575445fc6-cm9h6 1/1 Running 0 150m job-history-857564d575-tjmhh 1/1 Running 0 150m type-service-74c5cd66-vdqkd 1/1 Running 1 (149m ago) 150m toolbox-649f95d795-n76v6 1/1 Running 1 (149m ago) 150m scheduler-7f9567b8f4-zh4mp 1/1 Running 1 (149m ago) 150m file-storage-0 1/1 Running 0 149m file-broker-bdb9548b4-zzql2 1/1 Running 2 (149m ago) 150m ubuntu@chipster:~/git/chipster-openshift/k3s$ bash deploy.bash -f ~/values.yaml --set deployments.comp.configs.comp-max-jobs=\"3\" Error: UPGRADE FAILED: execution error at (chipster/templates/auth-secret.yaml:18:49): deployments.*.password is required ubuntu@chipster:~/git/chipster-openshift/k3s$ bash get-secret.bash comp Error from server (NotFound): secrets "comp" not found ubuntu@chipster:~/git/chipster-openshift/k3s$ kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE auth 1/1 1 1 22h service-locator 1/1 1 1 22h session-db 1/1 1 1 22h session-worker 1/1 1 1 22h web-server 1/1 1 1 22h backup 1/1 1 1 22h job-history 1/1 1 1 22h type-service 1/1 1 1 22h toolbox 1/1 1 1 22h scheduler 1/1 1 1 22h file-broker 1/1 1 1 22h This certainly triggers the same error if I try to add to ~/values.yaml deployments: comp: configs: comp-max-jobs: "3" Is there something easy to test to fix the issue or you will suggest starting from scratch? Best regards Pavlin Mitev ________________________________ Pavlin Mitev Uppsala Multidisciplinary Center for Advanced Computational Science (UPPMAX) Uppsala University, Box 337 SE-751 05 Uppsala, Sweden Tel: +46 18 471-3666 När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/ E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy |
From: Chipster <chi...@cs...> - 2023-04-26 13:55:41
|
Hi again, our Ansible playbook will now install a specific K3s version (now v1.26.4+k3s1), so the K3s version shouldn't anymore change by accident. I wrote some instructions for moving the data to hostPath volumes, which allows you to completely uninstall and reinstall K3s: https://github.com/chipster/chipster-openshift/blob/k3s/k3s/change-k3s-version.md I'm not aware of anything wrong with your current K3s version, so there shouldn't be any particular hurry to do this. I think this hostPath configuration makes a lot of sense for all single-node installations that will be maintained for a longer period of time, so I think it's good idea to configure this also right from the start if you setup new Chipster servers in the future. Regards, Petri > On 20. Apr 2023, at 19.52, Chipster <chi...@cs...> wrote: > > Hi Oli, > > >> On 14. Apr 2023, at 18.04, Oliver Heil <o....@dk...> wrote: >> >> Hi Petri, >> both chipsters are back again, thank you! >> (going back to k3s version v1.25.4 and doing the type-service solution on both) > > phew, I had completely missed your message so far, but I'm happy to hear that the servers came up finally! > > In the mean time I set up a Chipster behind a proxy. A new installation of the latest K3s v1.26.3 seemed to work. This was unnecessary after all for your case, but at least I managed to start cleaning up some Docker specific instructions that are not needed anymore, when K3s moved from Docker to containerd. > >> A general question and more like a criticism: >> chipster is incompatible with k3s version v1.25.8 but not with k3s version v1.25.4 ? >> Incompatibility with patch version change from 1.25.8 to 1.25.4 ? >> You know that I am not a fan of docker/kubernetes/... and this really strengthens my prejudices :-( > > Yeah, I'm disappointed too how K3s behaved in this update. I'll change the Ansible playbook to install a specific K3s version (v1.26.3+k3s1 to start with) so that we have a bit more control when and how K3s updates happen. > >> Another more constructive question: >> Why did I run into the type-service issue ? I didn't change anything on both machines, but over >> the easter weekend bot machines had this troubles. My understanding from last time was, that >> this may happen after the update routine from the docs, and not just from nowhere or just >> over time. > > I would guess that the type-service pods tried to restart on their own, and then failed because of this type-service issue. K3s restarts pods when those are not responding to their health checks. This could be caused for example if the process leaks memory or there is a temporary problem in network or other hardware resources. > > Anyway, our plan to add version tags to container images should make sure the container images won't update on thier own. Unfortunately we haven't had time to do this yet. Although, I'm not sure how this particular type-service problem will react, until we have figured out what is really causing it. > > >> Last question: >> When I now want to update ubuntu and everything according to the documents, can I go like: >> 1) do the update routine >> 2) after k3s update (from apt-get upgrade) go back to version 1.25.4 >> 3) continue update routine >> ? > > K3s was installed in the Ansible playbook, no with apt. So running "apt-get upgrade" should be safe. You could run the Ansible playbook and then downgrade K3s, Or you can skip the Ansible playbook altogether, there is not much to update in addition to that K3s. > > My colleague suggested that we should add a option for host mounting all important storage volumes (session-db-posgresql, file-storage). I think that's a great idea. This should allow you to reinstall K3s and Chipster and still keep the existing sessions. I'll try to see how this goes next. > > Best regards, > Petri > >> Well, of course, until something else pops up. >> Regards and have a nice weekend, >> Oli >> >> -- >> Oliver Heil >> Microarray Core Facility >> Bioinformatics >> German Cancer Research Center (DKFZ) >> Foundation under Public Law >> Im Neuenheimer Feld 580 >> 69120 Heidelberg >> Germany >> o....@dk... >> Support: www.dkfz.de/gpcf/support/ >> www.dkfz.de > <0Wa2n0NVqunnP1mx.jpg> >> >> >> Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich >> VAT-ID No.: DE143293537 >> _______________________________________________ >> Chipster-tech mailing list >> Chi...@li... >> https://lists.sourceforge.net/lists/listinfo/chipster-tech > > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Chipster <chi...@cs...> - 2023-04-20 16:54:00
|
Hi Oli, > On 14. Apr 2023, at 18.04, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > both chipsters are back again, thank you! > (going back to k3s version v1.25.4 and doing the type-service solution on both) phew, I had completely missed your message so far, but I'm happy to hear that the servers came up finally! In the mean time I set up a Chipster behind a proxy. A new installation of the latest K3s v1.26.3 seemed to work. This was unnecessary after all for your case, but at least I managed to start cleaning up some Docker specific instructions that are not needed anymore, when K3s moved from Docker to containerd. > A general question and more like a criticism: > chipster is incompatible with k3s version v1.25.8 but not with k3s version v1.25.4 ? > Incompatibility with patch version change from 1.25.8 to 1.25.4 ? > You know that I am not a fan of docker/kubernetes/... and this really strengthens my prejudices :-( Yeah, I'm disappointed too how K3s behaved in this update. I'll change the Ansible playbook to install a specific K3s version (v1.26.3+k3s1 to start with) so that we have a bit more control when and how K3s updates happen. > Another more constructive question: > Why did I run into the type-service issue ? I didn't change anything on both machines, but over > the easter weekend bot machines had this troubles. My understanding from last time was, that > this may happen after the update routine from the docs, and not just from nowhere or just > over time. I would guess that the type-service pods tried to restart on their own, and then failed because of this type-service issue. K3s restarts pods when those are not responding to their health checks. This could be caused for example if the process leaks memory or there is a temporary problem in network or other hardware resources. Anyway, our plan to add version tags to container images should make sure the container images won't update on thier own. Unfortunately we haven't had time to do this yet. Although, I'm not sure how this particular type-service problem will react, until we have figured out what is really causing it. > Last question: > When I now want to update ubuntu and everything according to the documents, can I go like: > 1) do the update routine > 2) after k3s update (from apt-get upgrade) go back to version 1.25.4 > 3) continue update routine > ? K3s was installed in the Ansible playbook, no with apt. So running "apt-get upgrade" should be safe. You could run the Ansible playbook and then downgrade K3s, Or you can skip the Ansible playbook altogether, there is not much to update in addition to that K3s. My colleague suggested that we should add a option for host mounting all important storage volumes (session-db-posgresql, file-storage). I think that's a great idea. This should allow you to reinstall K3s and Chipster and still keep the existing sessions. I'll try to see how this goes next. Best regards, Petri > Well, of course, until something else pops up. > Regards and have a nice weekend, > Oli > > -- > Oliver Heil > Microarray Core Facility > Bioinformatics > German Cancer Research Center (DKFZ) > Foundation under Public Law > Im Neuenheimer Feld 580 > 69120 Heidelberg > Germany > o....@dk... > Support: www.dkfz.de/gpcf/support/ > www.dkfz.de |
From: Oliver H. <o....@dk...> - 2023-04-14 15:05:20
|
Hi Petri, both chipsters are back again, thank you! (going back to k3s version v1.25.4 and doing the type-service solution on both) A general question and more like a criticism: chipster is incompatible with k3s version v1.25.8 but not with k3s version v1.25.4 ? Incompatibility with patch version change from 1.25.8 to 1.25.4 ? You know that I am not a fan of docker/kubernetes/... and this really strengthens my prejudices :-( Another more constructive question: Why did I run into the type-service issue ? I didn't change anything on both machines, but over the easter weekend bot machines had this troubles. My understanding from last time was, that this may happen after the update routine from the docs, and not just from nowhere or just over time. Last question: When I now want to update ubuntu and everything according to the documents, can I go like: 1) do the update routine 2) after k3s update (from apt-get upgrade) go back to version 1.25.4 3) continue update routine ? Well, of course, until something else pops up. Regards and have a nice weekend, Oli -- Oliver Heil Microarray Core Facility Bioinformatics German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany o....@dk... <mailto:o....@dk...> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> www.dkfz.de <http://www.dkfz.de> Research for a Life without Cancer Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich VAT-ID No.: DE143293537 |
From: Chipster <chi...@cs...> - 2023-04-14 12:49:51
|
Hi Oli, unfortunately I can't find any good leads from the log. I noticed that your original version was a actually a bit older, so maybe you could try to install that: curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.25.4+k3s1" sh - I guess your servers are still using the HTTP proxy configuration? Maybe I could try to setup a new test machine next week to see if that affects the situation in any way. Regards, Petri > On 13. Apr 2023, at 15.37, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > Am 13.04.2023 um 13:17 schrieb Chipster: >> Hi Oli, >> >> thanks for the coredns log. It can't reach Kubernetes API, which should be served by the K3s. So let's take a look at its log next: >> >> journalctl -u k3s > The log is quite big and cluttered. Before I miss something important, you > can download it here (.bz2): > https://www.dkfz.de/gpcf/extern/chipstervm1_journalctl_20230413.log.bz2 > >> >> My colleague suspected that this might be some compatibility issue between your original K3s version and the latest v1.26. You could reinstall the earlier version: >> >> sudo systemctl stop k3s >> curl -sfL https://get.k3s.io <https://get.k3s.io/> | INSTALL_K3S_VERSION="v1.25.8+k3s1" sh - > Seems not to work, it's just the same as before: > > >> ubuntu@chipstervm1:~$ kubectl get pod >> NAME READY STATUS RESTARTS AGE >> comp-job-2f0f2ec4-34c2-4e13-97b1-59218db0645c-hisat2-paired-end 0/1 Completed 0 37d >> comp-job-1c8f4a08-972a-4485-8377-442705a2617a-hisat2-paired-end 0/1 Completed 0 37d >> chipster-session-db-postgresql-0 1/1 Running 9 (9m33s ago) 128d >> chipster-auth-postgresql-0 1/1 Running 9 (9m33s ago) 128d >> chipster-job-history-postgresql-0 1/1 Running 9 (9m33s ago) 128d >> file-broker-888d995dd-kmsgp 0/1 CrashLoopBackOff 624 (3m12s ago) 2d4h >> type-service-77668d7f89-5bqt4 0/1 CrashLoopBackOff 628 (3m10s ago) 2d4h >> service-locator-84ff79985d-9gc8j 0/1 CrashLoopBackOff 623 (3m14s ago) 2d4h >> session-worker-64d5b97597-ctnt2 0/1 CrashLoopBackOff 623 (2m59s ago) 2d4h >> job-history-6d697c4dbf-6rhqc 0/1 CrashLoopBackOff 623 (2m55s ago) 2d4h >> session-db-66b9744c9-m27s2 0/1 CrashLoopBackOff 624 (2m57s ago) 2d4h >> web-server-7d8879c488-742ln 0/1 CrashLoopBackOff 623 (2m53s ago) 2d4h >> auth-7b96b64f6c-pvtbz 0/1 CrashLoopBackOff 625 (2m57s ago) 2d4h >> scheduler-66f947dfd7-kmtwp 0/1 CrashLoopBackOff 623 (2m47s ago) 2d4h >> file-storage-0 0/1 CrashLoopBackOff 691 (2m48s ago) 2d4h >> backup-b56f87f9d-xg9k7 0/1 CrashLoopBackOff 623 (2m45s ago) 2d4h >> toolbox-5cb75ff89b-dfxvd 0/1 CrashLoopBackOff 620 (2m35s ago) 2d4h > This error: >> E0412 09:37:43.940770 1743329 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:> the server is currently unable to handle the request > > has vanished now. > >> If that doesn't work, maybe check if the logs of this earlier version would be more helpful. > I would say no, not more helpful, just the same: > >> ubuntu@chipstervm1:~$ kubectl -n kube-system logs deployment/coredns >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API >> .:53 >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d >> CoreDNS-1.9.4 >> linux/amd64, go1.19.1, 1f0a41a >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.43.0.1:443/version" <https://10.43.0.1/version>: dial tcp 10.43.0.1:443: i/o timeout >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.43.0.1:443/version" <https://10.43.0.1/version>: dial tcp 10.43.0.1:443: i/o timeout >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.43.0.1:443/version" <https://10.43.0.1/version>: dial tcp 10.43.0.1:443: i/o timeout >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> ... > > >> >> If Chipster starts after this, you shouldn't run the Ansible playbook anymore, because that would reinstall the latest K3s version. I guess the next step then would be to install a new Chipster server and somehow move the customer data. However, I'm currently working on updating the old Postgres database, which probably is going to require new server installation too, so you may want to wait for that if it's difficult for you to move data between different Chipster servers. > Doesn't sound fun :-( > > Before I start a new installation on chipstervm1, do you see a way to bring chipstervm2 back > to a running state without reinstallation? See my other mail from today. chipstervm2 is just > in the same state with only type-service-... not coming up. No Updates done, so still the "old" > ansible/kubernetes. > >> >> In the previous message I mentioned about the possibility of removing and reinstalling K3s, but in hindsight that's probably a bad idea, because then the new K3s wouldn't know how to handle the data volumes in /mnt/data/k3s/storage. > Well, I have done it yesterday, but it didn't make things better or worse. Just the same. > it must be something different. > > Regards,, > > Oli > > > -- > Oliver Heil > Microarray Core Facility > Bioinformatics > > German Cancer Research Center (DKFZ) > Foundation under Public Law > Im Neuenheimer Feld 580 > 69120 Heidelberg > Germany > > o....@dk... <mailto:o....@dk...> > Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> > www.dkfz.de <http://www.dkfz.de/> > Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich > VAT-ID No.: DE143293537 > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... <mailto:Chi...@li...> > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Chipster <chi...@cs...> - 2023-04-14 12:42:12
|
Hi Oli, maybe check first the current K3s version, just in case: k3s -version The old instructions for pulling the latest container image should fix the type-service: https://sourceforge.net/p/chipster/mailman/message/37748558/ Regards, Petri > On 13. Apr 2023, at 11.04, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > I have the other machine in the same state as chipstervm1 was before I started > the update procedure to solve the issue. > > chipstervm2 is currently in: > > >> NAME READY STATUS RESTARTS AGE >> chipster-auth-postgresql-0 1/1 Running 1 (2d ago) 134d >> chipster-session-db-postgresql-0 1/1 Running 1 (2d ago) 134d >> chipster-job-history-postgresql-0 1/1 Running 1 (2d ago) 134d >> auth-79dcddf848-hn8dl 1/1 Running 2 (2d ago) 128d >> service-locator-6bf76f4769-x9f9h 1/1 Running 3 (2d ago) 128d >> session-db-6645fb5f6f-v68d5 1/1 Running 4 (2d ago) 128d >> file-storage-0 1/1 Running 2 (2d ago) 128d >> file-broker-55b9d957df-qdznd 1/1 Running 8 (2d ago) 128d >> web-server-7d9db56948-rtc6t 1/1 Running 5 128d >> job-history-64dd78d76d-5c6jc 1/1 Running 5 128d >> session-worker-579fd79d89-5dflj 1/1 Running 5 (2d ago) 128d >> backup-7b9bfb945f-9zgp6 1/1 Running 4 (2d ago) 128d >> toolbox-75f6648c94-4lzb7 1/1 Running 4 (2d ago) 128d >> scheduler-6f8b44c6f4-6pwst 1/1 Running 6 (2d ago) 128d >> type-service-cd8488cf6-md4d8 0/1 CrashLoopBackOff 571 (5m1s ago) 128d > > Ubuntu 20.04.5 LTS says: > > >> 39 updates can be applied immediately. > > which I did not yet. > > Perhaps we can try to bring this one back in a coordinated way first... > > ? > > Regards, > > Oli > > > -- > > Oliver Heil > Microarray Core Facility > Bioinformatics > > German Cancer Research Center (DKFZ) > Foundation under Public Law > Im Neuenheimer Feld 580 > 69120 Heidelberg > Germany > > o....@dk... <mailto:o....@dk...> > Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> > www.dkfz.de <http://www.dkfz.de/> > Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich > VAT-ID No.: DE143293537 > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... <mailto:Chi...@li...> > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Oliver H. <o....@dk...> - 2023-04-13 12:37:22
|
Hi Petri, Am 13.04.2023 um 13:17 schrieb Chipster: > Hi Oli, > > thanks for the coredns log. It can't reach Kubernetes API, which > should be served by the K3s. So let's take a look at its log next: > > journalctl -u k3s The log is quite big and cluttered. Before I miss something important, you can download it here (.bz2): https://www.dkfz.de/gpcf/extern/chipstervm1_journalctl_20230413.log.bz2 > > My colleague suspected that this might be some compatibility issue > between your original K3s version and the latest v1.26. You could > reinstall the earlier version: > > sudo systemctl stop k3s > curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.25.8+k3s1" sh - Seems not to work, it's just the same as before: > ubuntu@chipstervm1:~$ kubectl get pod > NAME READY STATUS RESTARTS AGE > comp-job-2f0f2ec4-34c2-4e13-97b1-59218db0645c-hisat2-paired-end > 0/1 Completed 0 37d > comp-job-1c8f4a08-972a-4485-8377-442705a2617a-hisat2-paired-end > 0/1 Completed 0 37d > chipster-session-db-postgresql-0 1/1 Running 9 (9m33s > ago) 128d > chipster-auth-postgresql-0 1/1 Running 9 (9m33s > ago) 128d > chipster-job-history-postgresql-0 1/1 Running 9 (9m33s > ago) 128d > file-broker-888d995dd-kmsgp 0/1 CrashLoopBackOff 624 (3m12s > ago) 2d4h > type-service-77668d7f89-5bqt4 0/1 CrashLoopBackOff 628 (3m10s > ago) 2d4h > service-locator-84ff79985d-9gc8j 0/1 CrashLoopBackOff 623 (3m14s > ago) 2d4h > session-worker-64d5b97597-ctnt2 0/1 CrashLoopBackOff 623 (2m59s > ago) 2d4h > job-history-6d697c4dbf-6rhqc 0/1 CrashLoopBackOff 623 (2m55s > ago) 2d4h > session-db-66b9744c9-m27s2 0/1 CrashLoopBackOff 624 (2m57s > ago) 2d4h > web-server-7d8879c488-742ln 0/1 CrashLoopBackOff 623 (2m53s > ago) 2d4h > auth-7b96b64f6c-pvtbz 0/1 CrashLoopBackOff 625 (2m57s ago) 2d4h > scheduler-66f947dfd7-kmtwp 0/1 CrashLoopBackOff 623 (2m47s > ago) 2d4h > file-storage-0 0/1 CrashLoopBackOff 691 (2m48s ago) 2d4h > backup-b56f87f9d-xg9k7 0/1 CrashLoopBackOff 623 (2m45s ago) 2d4h > toolbox-5cb75ff89b-dfxvd 0/1 CrashLoopBackOff 620 (2m35s ago) 2d4h This error: > E0412 09:37:43.940770 1743329 memcache.go:287] couldn't get resource > list formetrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:>the > server is currently unable to handle the request has vanished now. > If that doesn't work, maybe check if the logs of this earlier version > would be more helpful. I would say no, not more helpful, just the same: > ubuntu@chipstervm1:~$ kubectl -n kube-system logs deployment/coredns > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API > .:53 > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/reload: Running configuration SHA512 = > b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d > CoreDNS-1.9.4 > linux/amd64, go1.19.1, 1f0a41a > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get > "https://10.43.0.1:443/version": dial tcp 10.43.0.1:443: i/o timeout > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get > "https://10.43.0.1:443/version": dial tcp 10.43.0.1:443: i/o timeout > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get > "https://10.43.0.1:443/version": dial tcp 10.43.0.1:443: i/o timeout > [INFO] plugin/ready: Still waiting on: "kubernetes" > ... > > If Chipster starts after this, you shouldn't run the Ansible playbook > anymore, because that would reinstall the latest K3s version. I guess > the next step then would be to install a new Chipster server and > somehow move the customer data. However, I'm currently working on > updating the old Postgres database, which probably is going to require > new server installation too, so you may want to wait for that if it's > difficult for you to move data between different Chipster servers. Doesn't sound fun :-( Before I start a new installation on chipstervm1, do you see a way to bring chipstervm2 back to a running state without reinstallation? See my other mail from today. chipstervm2 is just in the same state with only type-service-... not coming up. No Updates done, so still the "old" ansible/kubernetes. > > In the previous message I mentioned about the possibility of removing > and reinstalling K3s, but in hindsight that's probably a bad idea, > because then the new K3s wouldn't know how to handle the data volumes > in /mnt/data/k3s/storage. Well, I have done it yesterday, but it didn't make things better or worse. Just the same. it must be something different. Regards,, Oli -- Oliver Heil Microarray Core Facility Bioinformatics German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany o....@dk... <mailto:o....@dk...> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> www.dkfz.de <http://www.dkfz.de> Research for a Life without Cancer Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich VAT-ID No.: DE143293537 |
From: Chipster <chi...@cs...> - 2023-04-13 11:18:32
|
Hi Oli, thanks for the coredns log. It can't reach Kubernetes API, which should be served by the K3s. So let's take a look at its log next: journalctl -u k3s My colleague suspected that this might be some compatibility issue between your original K3s version and the latest v1.26. You could reinstall the earlier version: sudo systemctl stop k3s curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.25.8+k3s1" sh - If that doesn't work, maybe check if the logs of this earlier version would be more helpful. If Chipster starts after this, you shouldn't run the Ansible playbook anymore, because that would reinstall the latest K3s version. I guess the next step then would be to install a new Chipster server and somehow move the customer data. However, I'm currently working on updating the old Postgres database, which probably is going to require new server installation too, so you may want to wait for that if it's difficult for you to move data between different Chipster servers. In the previous message I mentioned about the possibility of removing and reinstalling K3s, but in hindsight that's probably a bad idea, because then the new K3s wouldn't know how to handle the data volumes in /mnt/data/k3s/storage. Regards, Petri > On 12. Apr 2023, at 11.00, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > I did as you suggested but nothing changed. > > Here the log you requested (I snipped some repetitions): > > >> ubuntu@chipstervm1:~$ kubectl -n kube-system logs deployment/coredns >> E0412 09:37:43.940770 1743329 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:> the server is currently unable to handle the request >> E0412 09:37:43.970551 1743329 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:> the server is currently unable to handle the request >> E0412 09:37:43.975847 1743329 memcache.go:121] couldn't get resource list for metrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:> the server is currently unable to handle the request >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [INFO] plugin/kubernetes: waiting for Kubernetes API before starting server >> [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API >> .:53 >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/reload: Running configuration SHA512 = b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d >> CoreDNS-1.9.4 >> linux/amd64, go1.19.1, 1f0a41a >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.43.0.1:443/version" <https://10.43.0.1/version>: dial tcp 10.43.0.1:443: i/o timeout >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/ready: Still waiting on: "kubernetes" >> [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server >> [INFO] plugin/ready: Still waiting on: "kubernetes" > > > What do you think about this > >> E0412 09:37:43.940770 1743329 memcache.go:287] couldn't get resource list for metrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:> the server is currently unable to handle the request > > I see this for every "kubectl" command. > > I restarted, reinstalled, rebooted but it's the same as before. > > Regards, > > Oli > > > > -- > Oliver Heil > Microarray Core Facility > Bioinformatics > > German Cancer Research Center (DKFZ) > Foundation under Public Law > Im Neuenheimer Feld 580 > 69120 Heidelberg > Germany > > o....@dk... <mailto:o....@dk...> > Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> > www.dkfz.de <http://www.dkfz.de/> > Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich > VAT-ID No.: DE143293537 > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... <mailto:Chi...@li...> > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Oliver H. <o....@dk...> - 2023-04-13 08:04:49
|
Hi Petri, I have the other machine in the same state as chipstervm1 was before I started the update procedure to solve the issue. chipstervm2 is currently in: > NAME READY STATUS RESTARTS AGE > chipster-auth-postgresql-0 1/1 Running 1 (2d ago) 134d > chipster-session-db-postgresql-0 1/1 Running 1 (2d ago) 134d > chipster-job-history-postgresql-0 1/1 Running 1 (2d ago) 134d > auth-79dcddf848-hn8dl 1/1 Running 2 (2d ago) 128d > service-locator-6bf76f4769-x9f9h 1/1 Running 3 (2d ago) 128d > session-db-6645fb5f6f-v68d5 1/1 Running 4 (2d ago) 128d > file-storage-0 1/1 Running 2 (2d ago) 128d > file-broker-55b9d957df-qdznd 1/1 Running 8 (2d ago) 128d > web-server-7d9db56948-rtc6t 1/1 Running 5 128d > job-history-64dd78d76d-5c6jc 1/1 Running 5 128d > session-worker-579fd79d89-5dflj 1/1 Running 5 (2d ago) 128d > backup-7b9bfb945f-9zgp6 1/1 Running 4 (2d ago) 128d > toolbox-75f6648c94-4lzb7 1/1 Running 4 (2d ago) 128d > scheduler-6f8b44c6f4-6pwst 1/1 Running 6 (2d ago) 128d > type-service-cd8488cf6-md4d8 0/1 CrashLoopBackOff 571 (5m1s > ago) 128d Ubuntu 20.04.5 LTS says: > 39 updates can be applied immediately. which I did not yet. Perhaps we can try to bring this one back in a coordinated way first... ? Regards, Oli -- Oliver Heil Microarray Core Facility Bioinformatics German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany o....@dk... <mailto:o....@dk...> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> www.dkfz.de <http://www.dkfz.de> Research for a Life without Cancer Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich VAT-ID No.: DE143293537 |
From: Oliver H. <o....@dk...> - 2023-04-12 08:00:51
|
Hi Petri, I did as you suggested but nothing changed. Here the log you requested (I snipped some repetitions): > ubuntu@chipstervm1:~$ kubectl -n kube-system logs deployment/coredns > E0412 09:37:43.940770 1743329 memcache.go:287] couldn't get resource > list for metrics.k8s.io/v1beta1: the server is currently unable to > handle the request > E0412 09:37:43.970551 1743329 memcache.go:121] couldn't get resource > list for metrics.k8s.io/v1beta1: the server is currently unable to > handle the request > E0412 09:37:43.975847 1743329 memcache.go:121] couldn't get resource > list for metrics.k8s.io/v1beta1: the server is currently unable to > handle the request > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [INFO] plugin/kubernetes: waiting for Kubernetes API before starting > server > [WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API > .:53 > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/reload: Running configuration SHA512 = > b941b080e5322f6519009bb49349462c7ddb6317425b0f6a83e5451175b720703949e3f3b454a24e77f3ffe57fd5e9c6130e528a5a1dd00d9000e4afd6c1108d > CoreDNS-1.9.4 > linux/amd64, go1.19.1, 1f0a41a > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get > "https://10.43.0.1:443/version": dial tcp 10.43.0.1:443: i/o timeout > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/ready: Still waiting on: "kubernetes" > [WARNING] No files matching import glob pattern: > /etc/coredns/custom/*.server > [INFO] plugin/ready: Still waiting on: "kubernetes" What do you think about this > E0412 09:37:43.940770 1743329 memcache.go:287] couldn't get resource > list for metrics.k8s.io/v1beta1: the server is currently unable to > handle the request I see this for every "kubectl" command. I restarted, reinstalled, rebooted but it's the same as before. Regards, Oli -- Oliver Heil Microarray Core Facility Bioinformatics German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany o....@dk... <mailto:o....@dk...> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> www.dkfz.de <http://www.dkfz.de> Research for a Life without Cancer Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich VAT-ID No.: DE143293537 |
From: Chipster <chi...@cs...> - 2023-04-11 14:38:12
|
Hi Oli, both services complain about "UnknownHostException", so it looks like the internal DNS of the K3s isn't working at all. I don't think I have seen this before. K3s internal services are running in namespace "kube-system" and it seems to have a deployment called "coredns", which probably is responsible for this. Maybe you could take a look at its logs: kubectl -n kube-system logs deployment/coredns My logs have repeating warning "No files matching import glob pattern: /etc/coredns/custom/*.server", so apparently that's expected. I guess you could try to restart coredns anyway: kubectl rollout restart -n kube-system deployment/coredns Wait the old pod to disappear: watch kubectl get pod -n kube-system Maybe check the coredns logs again, and if you suspect that this may have helped, restart Chipster pods: bash restart.bash And see if the Chipster pods start now: watch kubectl get pod If that doesn't help, I would next try to run the Ansible playbook again to reinstall latest version of K3s: ansible-playbook ansible/install-deps.yml -i "localhost," -c local -e user=$(whoami) And reboot the server. If this doesn't help, then I would maybe first try to uninstall K3s and then try to install it again, but I haven't checked how to do it yet. Regards, Petri > On 11. Apr 2023, at 16.05, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > it seems there is a database not running: > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs deployment/auth >> [2023-04-11 11:37:54,380] INFO: load JAAS config from conf/jaas.config (in AuthenticationService:83) >> [2023-04-11 11:37:54,685] INFO: test connection to jdbc:postgresql://chipster-auth-postgresql.default.svc.cluster.local:5432/auth_db (in HibernateUtil:136) >> Exception in thread "main" java.lang.RuntimeException: failed to connect to jdbc:postgresql://chipster-auth-postgresql.default.svc.cluster.local:5432/auth_db >> at fi.csc.chipster.rest.hibernate.HibernateUtil.testConnection(HibernateUtil.java:153) >> at fi.csc.chipster.rest.hibernate.HibernateUtil.init(HibernateUtil.java:88) >> at fi.csc.chipster.rest.hibernate.HibernateUtil.<init>(HibernateUtil.java:72) >> at fi.csc.chipster.auth.AuthenticationService.startServer(AuthenticationService.java:87) >> at fi.csc.chipster.auth.AuthenticationService.main(AuthenticationService.java:149) >> Caused by: org.postgresql.util.PSQLException: The connection attempt failed. >> at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) >> at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) >> at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:263) >> at org.postgresql.Driver.makeConnection(Driver.java:443) >> at org.postgresql.Driver.connect(Driver.java:297) >> at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) >> at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189) >> at fi.csc.chipster.rest.hibernate.HibernateUtil.testConnection(HibernateUtil.java:141) >> ... 4 more >> Caused by: java.net.UnknownHostException: chipster-auth-postgresql.default.svc.cluster.local >> at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229) >> at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >> at java.base/java.net.Socket.connect(Socket.java:609) >> at org.postgresql.core.PGStream.createSocket(PGStream.java:243) >> at org.postgresql.core.PGStream.<init>(PGStream.java:98) >> at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:132) >> at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) >> ... 11 more > > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs deployment/service-locator >> [2023-04-11 11:39:25,448] INFO: get token from http://auth for service-locator (in AuthenticationClient:138) >> Exception in thread "main" jakarta.ws.rs.ProcessingException: java.net.UnknownHostException: auth >> at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:264) >> at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) >> at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:632) >> at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:654) >> at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:648) >> at org.glassfish.jersey.internal.Errors.process(Errors.java:292) >> at org.glassfish.jersey.internal.Errors.process(Errors.java:274) >> at org.glassfish.jersey.internal.Errors.process(Errors.java:205) >> at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) >> at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:648) >> at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:631) >> at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:434) >> at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:335) >> at fi.csc.chipster.auth.AuthenticationClient.getUserTokenFromAuth(AuthenticationClient.java:143) >> at fi.csc.chipster.auth.AuthenticationClient.construct(AuthenticationClient.java:102) >> at fi.csc.chipster.auth.AuthenticationClient.<init>(AuthenticationClient.java:89) >> at fi.csc.chipster.servicelocator.ServiceLocator.startServer(ServiceLocator.java:59) >> at fi.csc.chipster.servicelocator.ServiceLocator.main(ServiceLocator.java:128) >> Caused by: java.net.UnknownHostException: auth >> at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229) >> at java.base/java.net.Socket.connect(Socket.java:609) >> at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) >> at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:507) >> at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:602) >> at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:275) >> at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:374) >> at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:395) >> at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253) >> at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) >> at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) >> at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) >> at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592) >> at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) >> at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) >> at org.glassfish.jersey.client.internal.HttpUrlConnector.handleException(HttpUrlConnector.java:540) >> at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:370) >> at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:262) >> ... 17 more >> [2023-04-11 11:39:25,952] INFO: service-locator interrupted (in RestUtils$2:522) >> Exception in thread "Thread-0" java.lang.NullPointerException >> at fi.csc.chipster.rest.RestUtils$2.run(RestUtils.java:523) > > > Am 11.04.2023 um 13:30 schrieb Chipster: >> Hi Oli, >> >> I tried to install a new Chipster server and update it, but unfortunately it worked fine, so that didn't help to replicate this problem. >> >> However, I probably know the reason for your original problem in type-service. We are still relying on my weekly image builds to hide the type-service issue. I usually do it on Mondays, but this week I ran it only today at 9:25 (UTC+3) because of the Easter holidays. Maybe you managed to restart it already before that? Sorry for the trouble, I should have realised to run the build one extra time before heading to holidays. >> >> Would the logs tell why the pods are crashing? >> >> kubectl logs deployment/auth >> kubectl logs deployment/service-locator >> >> Regards, >> Petri >> > > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Oliver H. <o....@dk...> - 2023-04-11 13:05:13
|
Hi Petri, it seems there is a database not running: > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs > deployment/auth > [2023-04-11 11:37:54,380] INFO: load JAAS config from conf/jaas.config > (in AuthenticationService:83) > [2023-04-11 11:37:54,685] INFO: test connection to > jdbc:postgresql://chipster-auth-postgresql.default.svc.cluster.local:5432/auth_db > (in HibernateUtil:136) > Exception in thread "main" java.lang.RuntimeException: failed to > connect to > jdbc:postgresql://chipster-auth-postgresql.default.svc.cluster.local:5432/auth_db > at > fi.csc.chipster.rest.hibernate.HibernateUtil.testConnection(HibernateUtil.java:153) > at > fi.csc.chipster.rest.hibernate.HibernateUtil.init(HibernateUtil.java:88) > at > fi.csc.chipster.rest.hibernate.HibernateUtil.<init>(HibernateUtil.java:72) > at > fi.csc.chipster.auth.AuthenticationService.startServer(AuthenticationService.java:87) > at > fi.csc.chipster.auth.AuthenticationService.main(AuthenticationService.java:149) > Caused by: org.postgresql.util.PSQLException: The connection attempt > failed. > at > org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) > at > org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) > at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:263) > at org.postgresql.Driver.makeConnection(Driver.java:443) > at org.postgresql.Driver.connect(Driver.java:297) > at > java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) > at > java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189) > at > fi.csc.chipster.rest.hibernate.HibernateUtil.testConnection(HibernateUtil.java:141) > ... 4 more > Caused by: java.net.UnknownHostException: > chipster-auth-postgresql.default.svc.cluster.local > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at org.postgresql.core.PGStream.createSocket(PGStream.java:243) > at org.postgresql.core.PGStream.<init>(PGStream.java:98) > at > org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:132) > at > org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) > ... 11 more > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs > deployment/service-locator > [2023-04-11 11:39:25,448] INFO: get token from http://auth for > service-locator (in AuthenticationClient:138) > Exception in thread "main" jakarta.ws.rs.ProcessingException: > java.net.UnknownHostException: auth > at > org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:264) > at > org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) > at > org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:632) > at > org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:654) > at > org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:648) > at org.glassfish.jersey.internal.Errors.process(Errors.java:292) > at org.glassfish.jersey.internal.Errors.process(Errors.java:274) > at org.glassfish.jersey.internal.Errors.process(Errors.java:205) > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) > at > org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:648) > at > org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:631) > at > org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:434) > at > org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:335) > at > fi.csc.chipster.auth.AuthenticationClient.getUserTokenFromAuth(AuthenticationClient.java:143) > at > fi.csc.chipster.auth.AuthenticationClient.construct(AuthenticationClient.java:102) > at > fi.csc.chipster.auth.AuthenticationClient.<init>(AuthenticationClient.java:89) > at > fi.csc.chipster.servicelocator.ServiceLocator.startServer(ServiceLocator.java:59) > at > fi.csc.chipster.servicelocator.ServiceLocator.main(ServiceLocator.java:128) > Caused by: java.net.UnknownHostException: auth > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) > at > java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:507) > at > java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:602) > at > java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:275) > at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:374) > at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:395) > at > java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253) > at > java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) > at > java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) > at > java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) > at > java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592) > at > java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) > at > java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) > at > org.glassfish.jersey.client.internal.HttpUrlConnector.handleException(HttpUrlConnector.java:540) > at > org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:370) > at > org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:262) > ... 17 more > [2023-04-11 11:39:25,952] INFO: service-locator interrupted (in > RestUtils$2:522) > Exception in thread "Thread-0" java.lang.NullPointerException > at fi.csc.chipster.rest.RestUtils$2.run(RestUtils.java:523) Am 11.04.2023 um 13:30 schrieb Chipster: > Hi Oli, > > I tried to install a new Chipster server and update it, but > unfortunately it worked fine, so that didn't help to replicate this > problem. > > However, I probably know the reason for your original problem in > type-service. We are still relying on my weekly image builds to hide > the type-service issue. I usually do it on Mondays, but this week I > ran it only today at 9:25 (UTC+3) because of the Easter holidays. > Maybe you managed to restart it already before that? Sorry for the > trouble, I should have realised to run the build one extra time before > heading to holidays. > > Would the logs tell why the pods are crashing? > > kubectl logs deployment/auth > kubectl logs deployment/service-locator > > Regards, > Petri > |
From: Chipster <chi...@cs...> - 2023-04-11 11:47:16
|
Hi Oli, I tried to install a new Chipster server and update it, but unfortunately it worked fine, so that didn't help to replicate this problem. However, I probably know the reason for your original problem in type-service. We are still relying on my weekly image builds to hide the type-service issue. I usually do it on Mondays, but this week I ran it only today at 9:25 (UTC+3) because of the Easter holidays. Maybe you managed to restart it already before that? Sorry for the trouble, I should have realised to run the build one extra time before heading to holidays. Would the logs tell why the pods are crashing? kubectl logs deployment/auth kubectl logs deployment/service-locator Regards, Petri > On 11. Apr 2023, at 11.43, Oliver Heil <o....@dk...> wrote: > > Hi Petry, > > Same same as everytime: > > Chipster wasn't working with below problem: > > >> ubuntu@chipstervm1:~$ kubectl get pod >> NAME READY STATUS RESTARTS AGE >> comp-job-2f0f2ec4-34c2-4e13-97b1-59218db0645c-hisat2-paired-end 0/1 Completed 0 35d >> comp-job-1c8f4a08-972a-4485-8377-442705a2617a-hisat2-paired-end 0/1 Completed 0 35d >> chipster-session-db-postgresql-0 1/1 Running 6 (8m42s ago) 126d >> chipster-job-history-postgresql-0 1/1 Running 6 (8m42s ago) 126d >> chipster-auth-postgresql-0 1/1 Running 6 (8m42s ago) 126d >> auth-55cb7b9989-6728t 1/1 Running 82 (8m10s ago) 89d >> service-locator-7c5945f5b-mlvr2 1/1 Running 78 (7m48s ago) 89d >> session-worker-59f4f5d885-bd55q 1/1 Running 91 (7m46s ago) 89d >> web-server-574cc86c8d-57s9r 1/1 Running 98 (7m44s ago) 89d >> toolbox-d68f8ff88-9wq5z 1/1 Running 90 (7m38s ago) 89d >> backup-744cc78874-68p7w 1/1 Running 79 (7m26s ago) 89d >> session-db-749864bd99-4kv8s 1/1 Running 102 (7m15s ago) 89d >> job-history-75bd7679d5-mwjxk 1/1 Running 94 (6m40s ago) 89d >> file-storage-0 1/1 Running 101 (5m17s ago) 89d >> scheduler-5fff6c9f4c-qvlks 1/1 Running 104 (6m33s ago) 89d >> file-broker-5d5745d49d-rvv22 1/1 Running 94 (6m27s ago) 89d >> type-service-6968557565-rplkh 0/1 CrashLoopBackOff 1364 (2m30s ago) 89d >> ubuntu@chipstervm1:~$ >> ubuntu@chipstervm1:~$ kubectl logs type-service-6968557565-rplkh >> >> > start >> > node src/type-service.js >> >> npm ERR! code EACCES >> npm ERR! syscall open >> npm ERR! path /home/user/.npm/_cacache/tmp/e45d9325 >> npm ERR! errno -13 >> npm ERR! >> npm ERR! Your cache folder contains root-owned files, due to a bug in >> npm ERR! previous versions of npm which has since been addressed. >> npm ERR! >> npm ERR! To permanently fix this problem, please run: >> npm ERR! sudo chown -R 1000:0 "/home/user/.npm" >> >> npm ERR! Log files were not written due to an error writing to the directory: /home/user/.npm/_logs >> npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs in your terminal > > > So I first started the update routine according to > https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#Updates > > Last step I did was: > > >> bash deploy.bash -f ~/values.yaml --set image.localPullPolicy=Always > > It reported some error which is: > > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ bash deploy.bash -f ~/values.yaml --set image.localPullPolicy=Always >> E0411 10:37:20.647684 91511 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:> the server is currently unable to handle the request >> E0411 10:37:20.660001 91511 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: <http://metrics.k8s.io/v1beta1:> the server is currently unable to handle the request >> Release "chipster" has been upgraded. Happy Helming! >> NAME: chipster >> LAST DEPLOYED: Tue Apr 11 10:37:20 2023 >> NAMESPACE: default >> STATUS: deployed >> REVISION: 98 >> TEST SUITE: None >> NOTES: >> Following user accounts were configured >> ... > > Now I am waiting forever with: > > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pod >> NAME READY STATUS RESTARTS AGE >> comp-job-2f0f2ec4-34c2-4e13-97b1-59218db0645c-hisat2-paired-end 0/1 Completed 0 35d >> comp-job-1c8f4a08-972a-4485-8377-442705a2617a-hisat2-paired-end 0/1 Completed 0 35d >> chipster-job-history-postgresql-0 1/1 Running 7 (50m ago) 126d >> chipster-auth-postgresql-0 1/1 Running 7 (50m ago) 126d >> chipster-session-db-postgresql-0 1/1 Running 7 (50m ago) 126d >> auth-7b96b64f6c-pvtbz 0/1 CrashLoopBackOff 13 (4m50s ago) 47m >> scheduler-66f947dfd7-kmtwp 0/1 CrashLoopBackOff 13 (4m43s ago) 47m >> backup-b56f87f9d-xg9k7 0/1 CrashLoopBackOff 13 (4m22s ago) 47m >> file-broker-888d995dd-kmsgp 0/1 CrashLoopBackOff 13 (4m27s ago) 47m >> session-worker-64d5b97597-ctnt2 0/1 CrashLoopBackOff 13 (4m18s ago) 47m >> job-history-6d697c4dbf-6rhqc 0/1 CrashLoopBackOff 13 (4m18s ago) 47m >> service-locator-84ff79985d-9gc8j 0/1 CrashLoopBackOff 13 (4m16s ago) 47m >> session-db-66b9744c9-m27s2 0/1 CrashLoopBackOff 13 (3m58s ago) 47m >> type-service-77668d7f89-5bqt4 0/1 CrashLoopBackOff 13 (3m58s ago) 47m >> toolbox-5cb75ff89b-dfxvd 0/1 CrashLoopBackOff 13 (3m57s ago) 47m >> file-storage-0 0/1 CrashLoopBackOff 14 (3m7s ago) 46m >> web-server-7d8879c488-742ln 0/1 CrashLoopBackOff 13 (2m53s ago) 47m > > Any suggestions how to proceed? > > Regards, > > Oli > > > > -- > Oliver Heil > Microarray Core Facility > Bioinformatics > > German Cancer Research Center (DKFZ) > Foundation under Public Law > Im Neuenheimer Feld 580 > 69120 Heidelberg > Germany > > o....@dk... <mailto:o....@dk...> > Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> > www.dkfz.de <http://www.dkfz.de/> > Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich > VAT-ID No.: DE143293537 > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... <mailto:Chi...@li...> > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Oliver H. <o....@dk...> - 2023-04-11 08:43:26
|
Hi Petry, Same same as everytime: Chipster wasn't working with below problem: > ubuntu@chipstervm1:~$ kubectl get pod > NAME READY STATUS RESTARTS AGE > comp-job-2f0f2ec4-34c2-4e13-97b1-59218db0645c-hisat2-paired-end > 0/1 Completed 0 35d > comp-job-1c8f4a08-972a-4485-8377-442705a2617a-hisat2-paired-end > 0/1 Completed 0 35d > chipster-session-db-postgresql-0 1/1 Running 6 (8m42s > ago) 126d > chipster-job-history-postgresql-0 1/1 Running 6 (8m42s > ago) 126d > chipster-auth-postgresql-0 1/1 Running 6 (8m42s > ago) 126d > auth-55cb7b9989-6728t 1/1 Running 82 (8m10s ago) 89d > service-locator-7c5945f5b-mlvr2 1/1 Running 78 (7m48s > ago) 89d > session-worker-59f4f5d885-bd55q 1/1 Running 91 (7m46s > ago) 89d > web-server-574cc86c8d-57s9r 1/1 Running 98 (7m44s > ago) 89d > toolbox-d68f8ff88-9wq5z 1/1 Running 90 (7m38s ago) 89d > backup-744cc78874-68p7w 1/1 Running 79 (7m26s ago) 89d > session-db-749864bd99-4kv8s 1/1 Running 102 (7m15s > ago) 89d > job-history-75bd7679d5-mwjxk 1/1 Running 94 (6m40s > ago) 89d > file-storage-0 1/1 Running 101 (5m17s ago) 89d > scheduler-5fff6c9f4c-qvlks 1/1 Running 104 (6m33s > ago) 89d > file-broker-5d5745d49d-rvv22 1/1 Running 94 (6m27s > ago) 89d > type-service-6968557565-rplkh 0/1 CrashLoopBackOff 1364 (2m30s > ago) 89d > ubuntu@chipstervm1:~$ > ubuntu@chipstervm1:~$ kubectl logs type-service-6968557565-rplkh > > > start > > node src/type-service.js > > npm ERR! code EACCES > npm ERR! syscall open > npm ERR! path /home/user/.npm/_cacache/tmp/e45d9325 > npm ERR! errno -13 > npm ERR! > npm ERR! Your cache folder contains root-owned files, due to a bug in > npm ERR! previous versions of npm which has since been addressed. > npm ERR! > npm ERR! To permanently fix this problem, please run: > npm ERR! sudo chown -R 1000:0 "/home/user/.npm" > > npm ERR! Log files were not written due to an error writing to the > directory: /home/user/.npm/_logs > npm ERR! You can rerun the command with `--loglevel=verbose` to see > the logs in your terminal So I first started the update routine according to https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#Updates Last step I did was: > bash deploy.bash -f~/values.yaml --set image.localPullPolicy=Always It reported some error which is: > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ bash deploy.bash -f > ~/values.yaml --set image.localPullPolicy=Always > E0411 10:37:20.647684 91511 memcache.go:255] couldn't get resource > list for metrics.k8s.io/v1beta1: the server is currently unable to > handle the request > E0411 10:37:20.660001 91511 memcache.go:106] couldn't get resource > list for metrics.k8s.io/v1beta1: the server is currently unable to > handle the request > Release "chipster" has been upgraded. Happy Helming! > NAME: chipster > LAST DEPLOYED: Tue Apr 11 10:37:20 2023 > NAMESPACE: default > STATUS: deployed > REVISION: 98 > TEST SUITE: None > NOTES: > Following user accounts were configured > ... Now I am waiting forever with: > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pod > NAME READY STATUS RESTARTS AGE > comp-job-2f0f2ec4-34c2-4e13-97b1-59218db0645c-hisat2-paired-end > 0/1 Completed 0 35d > comp-job-1c8f4a08-972a-4485-8377-442705a2617a-hisat2-paired-end > 0/1 Completed 0 35d > chipster-job-history-postgresql-0 1/1 Running 7 (50m > ago) 126d > chipster-auth-postgresql-0 1/1 Running 7 (50m ago) > 126d > chipster-session-db-postgresql-0 1/1 Running 7 (50m > ago) 126d > auth-7b96b64f6c-pvtbz 0/1 CrashLoopBackOff 13 (4m50s ago) 47m > scheduler-66f947dfd7-kmtwp 0/1 CrashLoopBackOff 13 (4m43s ago) 47m > backup-b56f87f9d-xg9k7 0/1 CrashLoopBackOff 13 (4m22s ago) 47m > file-broker-888d995dd-kmsgp 0/1 CrashLoopBackOff 13 (4m27s > ago) 47m > session-worker-64d5b97597-ctnt2 0/1 CrashLoopBackOff 13 (4m18s > ago) 47m > job-history-6d697c4dbf-6rhqc 0/1 CrashLoopBackOff 13 (4m18s > ago) 47m > service-locator-84ff79985d-9gc8j 0/1 CrashLoopBackOff 13 (4m16s > ago) 47m > session-db-66b9744c9-m27s2 0/1 CrashLoopBackOff 13 (3m58s ago) 47m > type-service-77668d7f89-5bqt4 0/1 CrashLoopBackOff 13 (3m58s > ago) 47m > toolbox-5cb75ff89b-dfxvd 0/1 CrashLoopBackOff 13 (3m57s ago) 47m > file-storage-0 0/1 CrashLoopBackOff 14 (3m7s ago) 46m > web-server-7d8879c488-742ln 0/1 CrashLoopBackOff 13 (2m53s > ago) 47m Any suggestions how to proceed? Regards, Oli -- Oliver Heil Microarray Core Facility Bioinformatics German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany o....@dk... <mailto:o....@dk...> Support: www.dkfz.de/gpcf/support/ <https://www.dkfz.de/gpcf/support/> www.dkfz.de <http://www.dkfz.de> Research for a Life without Cancer Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich VAT-ID No.: DE143293537 |
From: Chipster <chi...@cs...> - 2023-01-02 06:41:36
|
Hi Oli, and one more correction. There was an unfortunate typo in my previous message. What I tried to say was "no" instead of "now": You have "hostPath" in the configuration, so there is _no_ automatic download... Best regards, Petri > On 23. Dec 2022, at 16.43, Chipster <chi...@cs...> wrote: > > Hi Oli, > > one more thing. The instructions in the previous message should fix the toolbox crashes. > > About the tools-bin download: You have "hostPath" in the configuration, so there is now automatic download, but you have done it manually: https://github.com/chipster/chipster-openshift/blob/k3s/k3s/tools-bin-host-mount.md <https://github.com/chipster/chipster-openshift/blob/k3s/k3s/tools-bin-host-mount.md> . The manual download is probably a good way to do it in the future too, because it's easier follow and debug in case anything goes wrong. > > Best regards, > Petri > >> On 19. Dec 2022, at 16.15, Oliver Heil <o....@dk... <mailto:o....@dk...>> wrote: >> >> Hi Petri, >> >> The toolbox pod is now created but stays in state CrashLoopBackOff. >> >> The download of >> >>> toolsBin: >>> version: chipster-4.5.16 >>> hostPath: /mnt/data/tools-bin >> isn't triggered whatever I do. >> >> Either it says folder chipster-4.5.16 doesn't exist and pod stays in ContainerCreating >> Or, if I create it beforehand, it keeps crashing with: >> >> >>> 2022-12-19 14:12:45,439] INFO: job factory: fi.csc.chipster.comp.java.JavaJobFactory (in RuntimeRepository:75) >>> [2022-12-19 14:12:45,440] INFO: tools-bin name: chipster-4.5.16 (in RuntimeRepository:76) >>> [2022-12-19 14:12:45,440] INFO: tools-bin path: /opt/chipster/tools (in RuntimeRepository:77) >>> [2022-12-19 14:12:45,441] INFO: load runtime R-4.1.1-fastqc (in RuntimeRepository:63) >>> Exception in thread "main" java.lang.IllegalArgumentException: configuration key not found: toolbox-runtime-command >>> at fi.csc.chipster.rest.Config.getDefault(Config.java:398) >>> at fi.csc.chipster.rest.Config.getString(Config.java:205) >>> at fi.csc.chipster.rest.Config.getString(Config.java:154) >>> at fi.csc.chipster.rest.Config.getString(Config.java:171) >>> at fi.csc.chipster.toolbox.runtime.RuntimeRepository.loadRuntimes(RuntimeRepository.java:65) >>> at fi.csc.chipster.toolbox.runtime.RuntimeRepository.<init>(RuntimeRepository.java:40) >>> at fi.csc.chipster.toolbox.ToolboxService.initialise(ToolboxService.java:107) >>> at fi.csc.chipster.toolbox.ToolboxService.<init>(ToolboxService.java:83) >>> at fi.csc.chipster.toolbox.ToolboxService.main(ToolboxService.java:261) >> >> >> Whatever triggers the download of tools-bin/chipster-4.5.16, it's not there for me. >> >> Regards, >> >> Oli >> >> >> >> Am 19.12.2022 um 10:44 schrieb Oliver Heil: >>> Hi Petri, >>> >>> any Ideas? >>> >>> Containers still in ContainerCreating state. >>> >>> I think it has to do with a missing place where I have to put some proxy >>> info into, but I have no idea where else I can put it in. >>> >>> Regards, >>> >>> Oli >>> >>> PS: no hurry >>> >>> Am 14.12.2022 um 13:40 schrieb Oliver Heil: >>>> Hi Petri, >>>> >>>> the type-service pod is running again, thanks! >>>> >>>> But the story unfolds again ;-) different as expected: >>>> >>>> Sidestory: I got an error with old tools-bin (4.5.2), but I guessed it is so because I am >>>> on master with chipster-tools but tools-bin is very old, so they don't match. Some of >>>> the R runtimes didn't start. (we can go back to 4.5.2 if you think this should be solved >>>> first). >>>> >>>> So I proceeded according to the update of tools-bin: >>>> >>>> As from the other server I edited the files >>>> >>>>> k3s/helm/chipster/templates/download-tools-bin-job.yaml >>>>> templates/jobs/download-tools-bin.bash >>>> >>>> and added the proxy information. >>>> tools.yaml now has: >>>> >>>> >>>>> toolsBin: >>>>> version: chipster-4.5.16 >>>>> hostPath: /mnt/data/tools-bin >>>> Doing >>>> >>>>> bash deploy.bash -f ~/values.yaml >>>> results in ever pending pvc: >>>> >>>> >>>> >>>>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pvc >>>>> NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE >>>>> data-chipster-auth-postgresql-0 Bound pvc-a3fb72a7-e6de-45f1-86c2-3770b5f3d47f 8Gi RWO local-path 489d >>>>> storage-file-storage-0 Bound pvc-25ac8422-0e05-4ec0-81b7-0ecaba423b04 300Gi RWO local-path 489d >>>>> data-chipster-job-history-postgresql-0 Bound pvc-6362bc18-d2a8-402c-870c-400fdbdb2273 8Gi RWO local-path 489d >>>>> data-chipster-session-db-postgresql-0 Bound pvc-b0dbba8a-84bc-42df-a8e4-b74df453efc6 8Gi RWO local-path 489d >>>>> tools-bin-chipster-4.5.16 Pending >>>> >>>> >>>>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pods >>>>> NAME READY STATUS RESTARTS AGE >>>>> chipster-session-db-postgresql-0 1/1 Running 1 (25h ago) 8d >>>>> chipster-job-history-postgresql-0 1/1 Running 1 (25h ago) 8d >>>>> chipster-auth-postgresql-0 1/1 Running 1 (25h ago) 8d >>>>> auth-6b77f48498-hzdx4 1/1 Running 0 21m >>>>> service-locator-65c86fb654-dsbr7 1/1 Running 1 (21m ago) 21m >>>>> web-server-d999cb748-vpn8c 1/1 Running 1 (21m ago) 21m >>>>> backup-7db5dbb8dc-8dl88 1/1 Running 1 (21m ago) 21m >>>>> session-db-6848844f49-jw6rq 1/1 Running 1 (21m ago) 21m >>>>> job-history-647c445bb-dg9w7 1/1 Running 1 (21m ago) 21m >>>>> type-service-7c5666c65c-djrm7 1/1 Running 2 (21m ago) 21m >>>>> file-storage-0 1/1 Running 0 21m >>>>> file-broker-f5896f6-m465v 1/1 Running 2 (21m ago) 21m >>>>> session-worker-79cbf8f456-ldfv2 1/1 Running 2 (21m ago) 21m >>>>> scheduler-5dc7d9c46b-j94c7 1/1 Running 2 (21m ago) 21m >>>>> toolbox-5bcf67cb54-sq8bt 0/1 ContainerCreating 0 14m >>>>> toolbox-79b98cdcf-dc82t 0/1 ContainerCreating 0 11m >>>> >>>> Yes, there are two toolboxes... !? >>>> A minor point, the docs say in >>>> >>>> https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package <https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package> >>>> >>>>> That will also will print you insctructions for how to follow the progress of the download and how to restart pods when it completes. >>>> Well, it doesn't. It tells only about the known >>>> >>>>> watch kubectl get pod >>>> which isn't what you want for the download progress. >>>> >>>> Anyways, my toolbox pods are created for ever it seems. Did I forgot a place for the proxy settings? >>>> Something else wrong? Of course no logs yet available for the two pods and no /mnt/data/tools-bin/chipster-4.5.16 >>>> is created. Perhaps I have to create it before manually? >>>> >>>> Regards, >>>> >>>> Oli >>>> >> >> _______________________________________________ >> Chipster-tech mailing list >> Chi...@li... <mailto:Chi...@li...> >> https://lists.sourceforge.net/lists/listinfo/chipster-tech > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Chipster <chi...@cs...> - 2022-12-23 14:43:36
|
Hi Oli, one more thing. The instructions in the previous message should fix the toolbox crashes. About the tools-bin download: You have "hostPath" in the configuration, so there is now automatic download, but you have done it manually: https://github.com/chipster/chipster-openshift/blob/k3s/k3s/tools-bin-host-mount.md <https://github.com/chipster/chipster-openshift/blob/k3s/k3s/tools-bin-host-mount.md> . The manual download is probably a good way to do it in the future too, because it's easier follow and debug in case anything goes wrong. Best regards, Petri > On 19. Dec 2022, at 16.15, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > The toolbox pod is now created but stays in state CrashLoopBackOff. > > The download of > >> toolsBin: >> version: chipster-4.5.16 >> hostPath: /mnt/data/tools-bin > isn't triggered whatever I do. > > Either it says folder chipster-4.5.16 doesn't exist and pod stays in ContainerCreating > Or, if I create it beforehand, it keeps crashing with: > > >> 2022-12-19 14:12:45,439] INFO: job factory: fi.csc.chipster.comp.java.JavaJobFactory (in RuntimeRepository:75) >> [2022-12-19 14:12:45,440] INFO: tools-bin name: chipster-4.5.16 (in RuntimeRepository:76) >> [2022-12-19 14:12:45,440] INFO: tools-bin path: /opt/chipster/tools (in RuntimeRepository:77) >> [2022-12-19 14:12:45,441] INFO: load runtime R-4.1.1-fastqc (in RuntimeRepository:63) >> Exception in thread "main" java.lang.IllegalArgumentException: configuration key not found: toolbox-runtime-command >> at fi.csc.chipster.rest.Config.getDefault(Config.java:398) >> at fi.csc.chipster.rest.Config.getString(Config.java:205) >> at fi.csc.chipster.rest.Config.getString(Config.java:154) >> at fi.csc.chipster.rest.Config.getString(Config.java:171) >> at fi.csc.chipster.toolbox.runtime.RuntimeRepository.loadRuntimes(RuntimeRepository.java:65) >> at fi.csc.chipster.toolbox.runtime.RuntimeRepository.<init>(RuntimeRepository.java:40) >> at fi.csc.chipster.toolbox.ToolboxService.initialise(ToolboxService.java:107) >> at fi.csc.chipster.toolbox.ToolboxService.<init>(ToolboxService.java:83) >> at fi.csc.chipster.toolbox.ToolboxService.main(ToolboxService.java:261) > > > Whatever triggers the download of tools-bin/chipster-4.5.16, it's not there for me. > > Regards, > > Oli > > > > Am 19.12.2022 um 10:44 schrieb Oliver Heil: >> Hi Petri, >> >> any Ideas? >> >> Containers still in ContainerCreating state. >> >> I think it has to do with a missing place where I have to put some proxy >> info into, but I have no idea where else I can put it in. >> >> Regards, >> >> Oli >> >> PS: no hurry >> >> Am 14.12.2022 um 13:40 schrieb Oliver Heil: >>> Hi Petri, >>> >>> the type-service pod is running again, thanks! >>> >>> But the story unfolds again ;-) different as expected: >>> >>> Sidestory: I got an error with old tools-bin (4.5.2), but I guessed it is so because I am >>> on master with chipster-tools but tools-bin is very old, so they don't match. Some of >>> the R runtimes didn't start. (we can go back to 4.5.2 if you think this should be solved >>> first). >>> >>> So I proceeded according to the update of tools-bin: >>> >>> As from the other server I edited the files >>> >>>> k3s/helm/chipster/templates/download-tools-bin-job.yaml >>>> templates/jobs/download-tools-bin.bash >>> >>> and added the proxy information. >>> tools.yaml now has: >>> >>> >>>> toolsBin: >>>> version: chipster-4.5.16 >>>> hostPath: /mnt/data/tools-bin >>> Doing >>> >>>> bash deploy.bash -f ~/values.yaml >>> results in ever pending pvc: >>> >>> >>> >>>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pvc >>>> NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE >>>> data-chipster-auth-postgresql-0 Bound pvc-a3fb72a7-e6de-45f1-86c2-3770b5f3d47f 8Gi RWO local-path 489d >>>> storage-file-storage-0 Bound pvc-25ac8422-0e05-4ec0-81b7-0ecaba423b04 300Gi RWO local-path 489d >>>> data-chipster-job-history-postgresql-0 Bound pvc-6362bc18-d2a8-402c-870c-400fdbdb2273 8Gi RWO local-path 489d >>>> data-chipster-session-db-postgresql-0 Bound pvc-b0dbba8a-84bc-42df-a8e4-b74df453efc6 8Gi RWO local-path 489d >>>> tools-bin-chipster-4.5.16 Pending >>> >>> >>>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pods >>>> NAME READY STATUS RESTARTS AGE >>>> chipster-session-db-postgresql-0 1/1 Running 1 (25h ago) 8d >>>> chipster-job-history-postgresql-0 1/1 Running 1 (25h ago) 8d >>>> chipster-auth-postgresql-0 1/1 Running 1 (25h ago) 8d >>>> auth-6b77f48498-hzdx4 1/1 Running 0 21m >>>> service-locator-65c86fb654-dsbr7 1/1 Running 1 (21m ago) 21m >>>> web-server-d999cb748-vpn8c 1/1 Running 1 (21m ago) 21m >>>> backup-7db5dbb8dc-8dl88 1/1 Running 1 (21m ago) 21m >>>> session-db-6848844f49-jw6rq 1/1 Running 1 (21m ago) 21m >>>> job-history-647c445bb-dg9w7 1/1 Running 1 (21m ago) 21m >>>> type-service-7c5666c65c-djrm7 1/1 Running 2 (21m ago) 21m >>>> file-storage-0 1/1 Running 0 21m >>>> file-broker-f5896f6-m465v 1/1 Running 2 (21m ago) 21m >>>> session-worker-79cbf8f456-ldfv2 1/1 Running 2 (21m ago) 21m >>>> scheduler-5dc7d9c46b-j94c7 1/1 Running 2 (21m ago) 21m >>>> toolbox-5bcf67cb54-sq8bt 0/1 ContainerCreating 0 14m >>>> toolbox-79b98cdcf-dc82t 0/1 ContainerCreating 0 11m >>> >>> Yes, there are two toolboxes... !? >>> A minor point, the docs say in >>> >>> https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package <https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package> >>> >>>> That will also will print you insctructions for how to follow the progress of the download and how to restart pods when it completes. >>> Well, it doesn't. It tells only about the known >>> >>>> watch kubectl get pod >>> which isn't what you want for the download progress. >>> >>> Anyways, my toolbox pods are created for ever it seems. Did I forgot a place for the proxy settings? >>> Something else wrong? Of course no logs yet available for the two pods and no /mnt/data/tools-bin/chipster-4.5.16 >>> is created. Perhaps I have to create it before manually? >>> >>> Regards, >>> >>> Oli >>> > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Chipster <chi...@cs...> - 2022-12-23 14:37:32
|
Hi Oli, yeah, it was me again. Thanks for reporting this. I was updating FastQC and I thought I could slip a small config change into chipster defaults without affecting others, but apparently I was too optimistic. I have now rebuilt the image and pulling it should fix this: sudo k3s crictl pull docker-registry.rahti.csc.fi/chipster-images/toolbox kubectl rollout restart deployment/toolbox watch kubectl get pod Sorry! Happy holidays, Petri > On 19. Dec 2022, at 16.15, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > The toolbox pod is now created but stays in state CrashLoopBackOff. > > The download of > >> toolsBin: >> version: chipster-4.5.16 >> hostPath: /mnt/data/tools-bin > isn't triggered whatever I do. > > Either it says folder chipster-4.5.16 doesn't exist and pod stays in ContainerCreating > Or, if I create it beforehand, it keeps crashing with: > > >> 2022-12-19 14:12:45,439] INFO: job factory: fi.csc.chipster.comp.java.JavaJobFactory (in RuntimeRepository:75) >> [2022-12-19 14:12:45,440] INFO: tools-bin name: chipster-4.5.16 (in RuntimeRepository:76) >> [2022-12-19 14:12:45,440] INFO: tools-bin path: /opt/chipster/tools (in RuntimeRepository:77) >> [2022-12-19 14:12:45,441] INFO: load runtime R-4.1.1-fastqc (in RuntimeRepository:63) >> Exception in thread "main" java.lang.IllegalArgumentException: configuration key not found: toolbox-runtime-command >> at fi.csc.chipster.rest.Config.getDefault(Config.java:398) >> at fi.csc.chipster.rest.Config.getString(Config.java:205) >> at fi.csc.chipster.rest.Config.getString(Config.java:154) >> at fi.csc.chipster.rest.Config.getString(Config.java:171) >> at fi.csc.chipster.toolbox.runtime.RuntimeRepository.loadRuntimes(RuntimeRepository.java:65) >> at fi.csc.chipster.toolbox.runtime.RuntimeRepository.<init>(RuntimeRepository.java:40) >> at fi.csc.chipster.toolbox.ToolboxService.initialise(ToolboxService.java:107) >> at fi.csc.chipster.toolbox.ToolboxService.<init>(ToolboxService.java:83) >> at fi.csc.chipster.toolbox.ToolboxService.main(ToolboxService.java:261) > > > Whatever triggers the download of tools-bin/chipster-4.5.16, it's not there for me. > > Regards, > > Oli > > > > Am 19.12.2022 um 10:44 schrieb Oliver Heil: >> Hi Petri, >> >> any Ideas? >> >> Containers still in ContainerCreating state. >> >> I think it has to do with a missing place where I have to put some proxy >> info into, but I have no idea where else I can put it in. >> >> Regards, >> >> Oli >> >> PS: no hurry >> >> Am 14.12.2022 um 13:40 schrieb Oliver Heil: >>> Hi Petri, >>> >>> the type-service pod is running again, thanks! >>> >>> But the story unfolds again ;-) different as expected: >>> >>> Sidestory: I got an error with old tools-bin (4.5.2), but I guessed it is so because I am >>> on master with chipster-tools but tools-bin is very old, so they don't match. Some of >>> the R runtimes didn't start. (we can go back to 4.5.2 if you think this should be solved >>> first). >>> >>> So I proceeded according to the update of tools-bin: >>> >>> As from the other server I edited the files >>> >>>> k3s/helm/chipster/templates/download-tools-bin-job.yaml >>>> templates/jobs/download-tools-bin.bash >>> >>> and added the proxy information. >>> tools.yaml now has: >>> >>> >>>> toolsBin: >>>> version: chipster-4.5.16 >>>> hostPath: /mnt/data/tools-bin >>> Doing >>> >>>> bash deploy.bash -f ~/values.yaml >>> results in ever pending pvc: >>> >>> >>> >>>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pvc >>>> NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE >>>> data-chipster-auth-postgresql-0 Bound pvc-a3fb72a7-e6de-45f1-86c2-3770b5f3d47f 8Gi RWO local-path 489d >>>> storage-file-storage-0 Bound pvc-25ac8422-0e05-4ec0-81b7-0ecaba423b04 300Gi RWO local-path 489d >>>> data-chipster-job-history-postgresql-0 Bound pvc-6362bc18-d2a8-402c-870c-400fdbdb2273 8Gi RWO local-path 489d >>>> data-chipster-session-db-postgresql-0 Bound pvc-b0dbba8a-84bc-42df-a8e4-b74df453efc6 8Gi RWO local-path 489d >>>> tools-bin-chipster-4.5.16 Pending >>> >>> >>>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pods >>>> NAME READY STATUS RESTARTS AGE >>>> chipster-session-db-postgresql-0 1/1 Running 1 (25h ago) 8d >>>> chipster-job-history-postgresql-0 1/1 Running 1 (25h ago) 8d >>>> chipster-auth-postgresql-0 1/1 Running 1 (25h ago) 8d >>>> auth-6b77f48498-hzdx4 1/1 Running 0 21m >>>> service-locator-65c86fb654-dsbr7 1/1 Running 1 (21m ago) 21m >>>> web-server-d999cb748-vpn8c 1/1 Running 1 (21m ago) 21m >>>> backup-7db5dbb8dc-8dl88 1/1 Running 1 (21m ago) 21m >>>> session-db-6848844f49-jw6rq 1/1 Running 1 (21m ago) 21m >>>> job-history-647c445bb-dg9w7 1/1 Running 1 (21m ago) 21m >>>> type-service-7c5666c65c-djrm7 1/1 Running 2 (21m ago) 21m >>>> file-storage-0 1/1 Running 0 21m >>>> file-broker-f5896f6-m465v 1/1 Running 2 (21m ago) 21m >>>> session-worker-79cbf8f456-ldfv2 1/1 Running 2 (21m ago) 21m >>>> scheduler-5dc7d9c46b-j94c7 1/1 Running 2 (21m ago) 21m >>>> toolbox-5bcf67cb54-sq8bt 0/1 ContainerCreating 0 14m >>>> toolbox-79b98cdcf-dc82t 0/1 ContainerCreating 0 11m >>> >>> Yes, there are two toolboxes... !? >>> A minor point, the docs say in >>> >>> https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package <https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package> >>> >>>> That will also will print you insctructions for how to follow the progress of the download and how to restart pods when it completes. >>> Well, it doesn't. It tells only about the known >>> >>>> watch kubectl get pod >>> which isn't what you want for the download progress. >>> >>> Anyways, my toolbox pods are created for ever it seems. Did I forgot a place for the proxy settings? >>> Something else wrong? Of course no logs yet available for the two pods and no /mnt/data/tools-bin/chipster-4.5.16 >>> is created. Perhaps I have to create it before manually? >>> >>> Regards, >>> >>> Oli >>> > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Oliver H. <o....@dk...> - 2022-12-19 14:15:24
|
Hi Petri, The toolbox pod is now created but stays in state CrashLoopBackOff. The download of > toolsBin: > version: chipster-4.5.16 > hostPath: /mnt/data/tools-bin isn't triggered whatever I do. Either it says folder chipster-4.5.16 doesn't exist and pod stays in ContainerCreating Or, if I create it beforehand, it keeps crashing with: > 2022-12-19 14:12:45,439] INFO: job factory: > fi.csc.chipster.comp.java.JavaJobFactory (in RuntimeRepository:75) > [2022-12-19 14:12:45,440] INFO: tools-bin name: chipster-4.5.16 (in > RuntimeRepository:76) > [2022-12-19 14:12:45,440] INFO: tools-bin path: /opt/chipster/tools > (in RuntimeRepository:77) > [2022-12-19 14:12:45,441] INFO: load runtime R-4.1.1-fastqc (in > RuntimeRepository:63) > Exception in thread "main" java.lang.IllegalArgumentException: > configuration key not found: toolbox-runtime-command > at fi.csc.chipster.rest.Config.getDefault(Config.java:398) > at fi.csc.chipster.rest.Config.getString(Config.java:205) > at fi.csc.chipster.rest.Config.getString(Config.java:154) > at fi.csc.chipster.rest.Config.getString(Config.java:171) > at > fi.csc.chipster.toolbox.runtime.RuntimeRepository.loadRuntimes(RuntimeRepository.java:65) > at > fi.csc.chipster.toolbox.runtime.RuntimeRepository.<init>(RuntimeRepository.java:40) > at > fi.csc.chipster.toolbox.ToolboxService.initialise(ToolboxService.java:107) > at > fi.csc.chipster.toolbox.ToolboxService.<init>(ToolboxService.java:83) > at > fi.csc.chipster.toolbox.ToolboxService.main(ToolboxService.java:261) Whatever triggers the download of tools-bin/chipster-4.5.16, it's not there for me. Regards, Oli Am 19.12.2022 um 10:44 schrieb Oliver Heil: > > Hi Petri, > > any Ideas? > > Containers still in ContainerCreating state. > > I think it has to do with a missing place where I have to put some proxy > info into, but I have no idea where else I can put it in. > > Regards, > > Oli > > PS: no hurry > > Am 14.12.2022 um 13:40 schrieb Oliver Heil: >> >> Hi Petri, >> >> the type-service pod is running again, thanks! >> >> But the story unfolds again ;-) different as expected: >> >> Sidestory: I got an error with old tools-bin (4.5.2), but I guessed >> it is so because I am >> on master with chipster-tools but tools-bin is very old, so they >> don't match. Some of >> the R runtimes didn't start. (we can go back to 4.5.2 if you think >> this should be solved >> first). >> >> So I proceeded according to the update of tools-bin: >> >> As from the other server I edited the files >> >>> k3s/helm/chipster/templates/download-tools-bin-job.yaml >>> templates/jobs/download-tools-bin.bash >> >> and added the proxy information. >> >> tools.yaml now has: >> >>> toolsBin: >>> version: chipster-4.5.16 >>> hostPath: /mnt/data/tools-bin >> >> Doing >> >>> bash deploy.bash -f~/values.yaml >> >> results in ever pending pvc: >> >> >>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pvc >>> NAME STATUS >>> VOLUME CAPACITY ACCESS MODES >>> STORAGECLASS AGE >>> data-chipster-auth-postgresql-0 Bound >>> pvc-a3fb72a7-e6de-45f1-86c2-3770b5f3d47f 8Gi RWO >>> local-path 489d >>> storage-file-storage-0 Bound >>> pvc-25ac8422-0e05-4ec0-81b7-0ecaba423b04 300Gi RWO >>> local-path 489d >>> data-chipster-job-history-postgresql-0 Bound >>> pvc-6362bc18-d2a8-402c-870c-400fdbdb2273 8Gi RWO >>> local-path 489d >>> data-chipster-session-db-postgresql-0 Bound >>> pvc-b0dbba8a-84bc-42df-a8e4-b74df453efc6 8Gi RWO >>> local-path 489d >>> tools-bin-chipster-4.5.16 Pending >> >> >>> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pods >>> NAME READY STATUS >>> RESTARTS AGE >>> chipster-session-db-postgresql-0 1/1 Running 1 (25h >>> ago) 8d >>> chipster-job-history-postgresql-0 1/1 Running 1 (25h >>> ago) 8d >>> chipster-auth-postgresql-0 1/1 Running 1 (25h >>> ago) 8d >>> auth-6b77f48498-hzdx4 1/1 Running >>> 0 21m >>> service-locator-65c86fb654-dsbr7 1/1 Running 1 (21m >>> ago) 21m >>> web-server-d999cb748-vpn8c 1/1 Running 1 (21m >>> ago) 21m >>> backup-7db5dbb8dc-8dl88 1/1 Running 1 (21m >>> ago) 21m >>> session-db-6848844f49-jw6rq 1/1 Running 1 (21m >>> ago) 21m >>> job-history-647c445bb-dg9w7 1/1 Running 1 (21m >>> ago) 21m >>> type-service-7c5666c65c-djrm7 1/1 Running 2 (21m >>> ago) 21m >>> file-storage-0 1/1 Running >>> 0 21m >>> file-broker-f5896f6-m465v 1/1 Running 2 (21m >>> ago) 21m >>> session-worker-79cbf8f456-ldfv2 1/1 Running 2 (21m >>> ago) 21m >>> scheduler-5dc7d9c46b-j94c7 1/1 Running 2 (21m >>> ago) 21m >>> toolbox-5bcf67cb54-sq8bt 0/1 ContainerCreating >>> 0 14m >>> toolbox-79b98cdcf-dc82t 0/1 ContainerCreating >>> 0 11m >> >> Yes, there are two toolboxes... !? >> >> A minor point, the docs say in >> >> https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package >> >>> That will also will print you insctructions for how to follow the >>> progress of the download and how to restart pods when it completes. >> >> Well, it doesn't. It tells only about the known >> >>> watch kubectl get pod >> >> which isn't what you want for the download progress. >> >> Anyways, my toolbox pods are created for ever it seems. Did I forgot >> a place for the proxy settings? >> Something else wrong? Of course no logs yet available for the two >> pods and no /mnt/data/tools-bin/chipster-4.5.16 >> is created. Perhaps I have to create it before manually? >> >> Regards, >> >> Oli >> |
From: Oliver H. <o....@dk...> - 2022-12-19 09:44:35
|
Hi Petri, any Ideas? Containers still in ContainerCreating state. I think it has to do with a missing place where I have to put some proxy info into, but I have no idea where else I can put it in. Regards, Oli PS: no hurry Am 14.12.2022 um 13:40 schrieb Oliver Heil: > > Hi Petri, > > the type-service pod is running again, thanks! > > But the story unfolds again ;-) different as expected: > > Sidestory: I got an error with old tools-bin (4.5.2), but I guessed it > is so because I am > on master with chipster-tools but tools-bin is very old, so they don't > match. Some of > the R runtimes didn't start. (we can go back to 4.5.2 if you think > this should be solved > first). > > So I proceeded according to the update of tools-bin: > > As from the other server I edited the files > >> k3s/helm/chipster/templates/download-tools-bin-job.yaml >> templates/jobs/download-tools-bin.bash > > and added the proxy information. > > tools.yaml now has: > >> toolsBin: >> version: chipster-4.5.16 >> hostPath: /mnt/data/tools-bin > > Doing > >> bash deploy.bash -f~/values.yaml > > results in ever pending pvc: > > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pvc >> NAME STATUS >> VOLUME CAPACITY ACCESS MODES >> STORAGECLASS AGE >> data-chipster-auth-postgresql-0 Bound >> pvc-a3fb72a7-e6de-45f1-86c2-3770b5f3d47f 8Gi RWO >> local-path 489d >> storage-file-storage-0 Bound >> pvc-25ac8422-0e05-4ec0-81b7-0ecaba423b04 300Gi RWO >> local-path 489d >> data-chipster-job-history-postgresql-0 Bound >> pvc-6362bc18-d2a8-402c-870c-400fdbdb2273 8Gi RWO >> local-path 489d >> data-chipster-session-db-postgresql-0 Bound >> pvc-b0dbba8a-84bc-42df-a8e4-b74df453efc6 8Gi RWO >> local-path 489d >> tools-bin-chipster-4.5.16 Pending > > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pods >> NAME READY STATUS >> RESTARTS AGE >> chipster-session-db-postgresql-0 1/1 Running 1 (25h >> ago) 8d >> chipster-job-history-postgresql-0 1/1 Running 1 (25h >> ago) 8d >> chipster-auth-postgresql-0 1/1 Running 1 (25h >> ago) 8d >> auth-6b77f48498-hzdx4 1/1 Running >> 0 21m >> service-locator-65c86fb654-dsbr7 1/1 Running 1 (21m >> ago) 21m >> web-server-d999cb748-vpn8c 1/1 Running 1 (21m >> ago) 21m >> backup-7db5dbb8dc-8dl88 1/1 Running 1 (21m >> ago) 21m >> session-db-6848844f49-jw6rq 1/1 Running 1 (21m >> ago) 21m >> job-history-647c445bb-dg9w7 1/1 Running 1 (21m >> ago) 21m >> type-service-7c5666c65c-djrm7 1/1 Running 2 (21m >> ago) 21m >> file-storage-0 1/1 Running >> 0 21m >> file-broker-f5896f6-m465v 1/1 Running 2 (21m >> ago) 21m >> session-worker-79cbf8f456-ldfv2 1/1 Running 2 (21m >> ago) 21m >> scheduler-5dc7d9c46b-j94c7 1/1 Running 2 (21m >> ago) 21m >> toolbox-5bcf67cb54-sq8bt 0/1 ContainerCreating >> 0 14m >> toolbox-79b98cdcf-dc82t 0/1 ContainerCreating >> 0 11m > > Yes, there are two toolboxes... !? > > A minor point, the docs say in > > https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package > >> That will also will print you insctructions for how to follow the >> progress of the download and how to restart pods when it completes. > > Well, it doesn't. It tells only about the known > >> watch kubectl get pod > > which isn't what you want for the download progress. > > Anyways, my toolbox pods are created for ever it seems. Did I forgot a > place for the proxy settings? > Something else wrong? Of course no logs yet available for the two pods > and no /mnt/data/tools-bin/chipster-4.5.16 > is created. Perhaps I have to create it before manually? > > Regards, > > Oli > |
From: Oliver H. <o....@dk...> - 2022-12-14 12:40:38
|
Hi Petri, the type-service pod is running again, thanks! But the story unfolds again ;-) different as expected: Sidestory: I got an error with old tools-bin (4.5.2), but I guessed it is so because I am on master with chipster-tools but tools-bin is very old, so they don't match. Some of the R runtimes didn't start. (we can go back to 4.5.2 if you think this should be solved first). So I proceeded according to the update of tools-bin: As from the other server I edited the files > k3s/helm/chipster/templates/download-tools-bin-job.yaml > templates/jobs/download-tools-bin.bash and added the proxy information. tools.yaml now has: > toolsBin: > version: chipster-4.5.16 > hostPath: /mnt/data/tools-bin Doing > bash deploy.bash -f~/values.yaml results in ever pending pvc: > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pvc > NAME STATUS > VOLUME CAPACITY ACCESS MODES > STORAGECLASS AGE > data-chipster-auth-postgresql-0 Bound > pvc-a3fb72a7-e6de-45f1-86c2-3770b5f3d47f 8Gi RWO > local-path 489d > storage-file-storage-0 Bound > pvc-25ac8422-0e05-4ec0-81b7-0ecaba423b04 300Gi RWO > local-path 489d > data-chipster-job-history-postgresql-0 Bound > pvc-6362bc18-d2a8-402c-870c-400fdbdb2273 8Gi RWO > local-path 489d > data-chipster-session-db-postgresql-0 Bound > pvc-b0dbba8a-84bc-42df-a8e4-b74df453efc6 8Gi RWO > local-path 489d > tools-bin-chipster-4.5.16 Pending > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pods > NAME READY STATUS > RESTARTS AGE > chipster-session-db-postgresql-0 1/1 Running 1 (25h > ago) 8d > chipster-job-history-postgresql-0 1/1 Running 1 (25h > ago) 8d > chipster-auth-postgresql-0 1/1 Running 1 (25h > ago) 8d > auth-6b77f48498-hzdx4 1/1 Running > 0 21m > service-locator-65c86fb654-dsbr7 1/1 Running 1 (21m > ago) 21m > web-server-d999cb748-vpn8c 1/1 Running 1 (21m > ago) 21m > backup-7db5dbb8dc-8dl88 1/1 Running 1 (21m > ago) 21m > session-db-6848844f49-jw6rq 1/1 Running 1 (21m > ago) 21m > job-history-647c445bb-dg9w7 1/1 Running 1 (21m > ago) 21m > type-service-7c5666c65c-djrm7 1/1 Running 2 (21m > ago) 21m > file-storage-0 1/1 Running > 0 21m > file-broker-f5896f6-m465v 1/1 Running 2 (21m > ago) 21m > session-worker-79cbf8f456-ldfv2 1/1 Running 2 (21m > ago) 21m > scheduler-5dc7d9c46b-j94c7 1/1 Running 2 (21m > ago) 21m > toolbox-5bcf67cb54-sq8bt 0/1 ContainerCreating > 0 14m > toolbox-79b98cdcf-dc82t 0/1 ContainerCreating > 0 11m Yes, there are two toolboxes... !? A minor point, the docs say in https://github.com/chipster/chipster-openshift/blob/master/k3s/README.md#download-the-tools-bin-package > That will also will print you insctructions for how to follow the > progress of the download and how to restart pods when it completes. Well, it doesn't. It tells only about the known > watch kubectl get pod which isn't what you want for the download progress. Anyways, my toolbox pods are created for ever it seems. Did I forgot a place for the proxy settings? Something else wrong? Of course no logs yet available for the two pods and no /mnt/data/tools-bin/chipster-4.5.16 is created. Perhaps I have to create it before manually? Regards, Oli Am 14.12.2022 um 10:56 schrieb Chipster: > Hi Oli, > > sorry, my workaround of manually rebuilding the image every Monday > didn't work very well when I was away. I built it now, so pulling the > new image should help again > (https://sourceforge.net/p/chipster/mailman/message/37738955/): > > sudo k3s crictl pull > docker-registry.rahti.csc.fi/chipster-images/chipster-web-server-js > <http://docker-registry.rahti.csc.fi/chipster-images/chipster-web-server-js> > http://docker-registry.rahti.csc.fi/chipster-images/chipster-web-server-js > > > kubectl rollout restart deployment/type-service > > # wait until the the old type-service pod has disappeared, then quit > with Ctrl+C > watch kubectl get pod > > Although, this doesn't explain why the toolbox is crashing. Let's hope > it heals when your tools-bin download is ready. > > Hope this helps, > Petri |
From: Chipster <chi...@cs...> - 2022-12-14 09:56:41
|
Hi Oli, sorry, my workaround of manually rebuilding the image every Monday didn't work very well when I was away. I built it now, so pulling the new image should help again (https://sourceforge.net/p/chipster/mailman/message/37738955/ <https://sourceforge.net/p/chipster/mailman/message/37738955/>): sudo k3s crictl pull docker-registry.rahti.csc.fi/chipster-images/chipster-web-server-js http://docker-registry.rahti.csc.fi/chipster-images/chipster-web-server-js <http://docker-registry.rahti.csc.fi/chipster-images/chipster-web-server-js> kubectl rollout restart deployment/type-service # wait until the the old type-service pod has disappeared, then quit with Ctrl+C watch kubectl get pod Although, this doesn't explain why the toolbox is crashing. Let's hope it heals when your tools-bin download is ready. Hope this helps, Petri > On 13. Dec 2022, at 13.46, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > here I am back again, not like the pods ;-) > > After going into the update routine just before I planed to upgrade > to the newest tool version (which takes probably longer), two of the pods > do not come back again. > > The last time you solved magically by a new container(?) in the repository, > so we didn't solved it actually. > > This is the current state: > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pod >> NAME READY STATUS RESTARTS AGE >> chipster-session-db-postgresql-0 1/1 Running 1 (63m ago) 7d22h >> chipster-job-history-postgresql-0 1/1 Running 1 (63m ago) 7d22h >> chipster-auth-postgresql-0 1/1 Running 1 (63m ago) 7d22h >> auth-688c6946d-ssxl9 1/1 Running 0 32m >> service-locator-787cf87685-97bpq 1/1 Running 2 (31m ago) 32m >> session-worker-586f48cc8f-mrnwd 1/1 Running 3 (31m ago) 32m >> backup-5bb7dd649f-gbmgj 1/1 Running 3 (31m ago) 32m >> web-server-778b9f9f46-6x69x 1/1 Running 3 (31m ago) 32m >> session-db-7f49fb79-4nm5z 1/1 Running 3 (31m ago) 32m >> job-history-6675d8c7b4-xlbdp 1/1 Running 3 (31m ago) 32m >> scheduler-5f4f78b754-9xbrw 1/1 Running 3 (31m ago) 32m >> file-storage-0 1/1 Running 3 (30m ago) 31m >> file-broker-dc7c86d5f-fgg8m 1/1 Running 4 (30m ago) 32m >> type-service-8489f88fb-54r4s 0/1 CrashLoopBackOff 11 (16s ago) 32m >> toolbox-6b9f9b8d64-v6wvl 0/1 CrashLoopBackOff 11 (7s ago) 32m > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs type-service-8489f88fb-54r4s >> >> > start >> > node src/type-service.js >> >> npm ERR! code EACCES >> npm ERR! syscall open >> npm ERR! path /home/user/.npm/_cacache/tmp/b7d792dd >> npm ERR! errno -13 >> npm ERR! >> npm ERR! Your cache folder contains root-owned files, due to a bug in >> npm ERR! previous versions of npm which has since been addressed. >> npm ERR! >> npm ERR! To permanently fix this problem, please run: >> npm ERR! sudo chown -R 1000:0 "/home/user/.npm" >> >> npm ERR! Log files were not written due to an error writing to the directory: /home/user/.npm/_logs >> npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs in your terminal > > > I am not able to open a bash inside the type-service, always saying > e.g. error: unable to upgrade connection: container not found ("type-service") > > I tried "type-service-8489f88fb-54r4s", "type-service" and others. Always "not found". > > Any suggestions? > > Regards, > > Oli > > > > > > > > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Oliver H. <o....@dk...> - 2022-12-14 09:52:27
|
Hi Eija, no problem for me. This server is now in "maintenance" mode for a timely lengthy procedure (updating tools), so there is no need for speed. Hopefully good recovery to all who are ill now! Regards, Oli Am 14.12.2022 um 07:12 schrieb chi...@cs...: > Hi Oliver, > > Sorry to hear that there is a problem again with the pods. Unfortunately many people are out of office now (Finland seems to have a wave of covid again), so it will be a little while before we can provide technical help. > > Best, > Eija > > ----- Original Message ----- > From: "Oliver Heil"<o....@dk...> > To: "chipster-tech"<chi...@li...> > Sent: Tuesday, 13 December, 2022 13:46:00 > Subject: [Chipster-tech] pods not coming back again > > Hi Petri, > > here I am back again, not like the pods ;-) > > After going into the update routine just before I planed to upgrade > to the newest tool version (which takes probably longer), two of the pods > do not come back again. > > The last time you solved magically by a new container(?) in the repository, > so we didn't solved it actually. > > This is the current state: > >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pod >> NAME READY STATUS RESTARTS AGE >> chipster-session-db-postgresql-0 1/1 Running 1 (63m ago) >> 7d22h >> chipster-job-history-postgresql-0 1/1 Running 1 (63m ago) >> 7d22h >> chipster-auth-postgresql-0 1/1 Running 1 (63m ago) >> 7d22h >> auth-688c6946d-ssxl9 1/1 Running 0 32m >> service-locator-787cf87685-97bpq 1/1 Running 2 (31m ago) 32m >> session-worker-586f48cc8f-mrnwd 1/1 Running 3 (31m ago) 32m >> backup-5bb7dd649f-gbmgj 1/1 Running 3 (31m ago) 32m >> web-server-778b9f9f46-6x69x 1/1 Running 3 (31m ago) 32m >> session-db-7f49fb79-4nm5z 1/1 Running 3 (31m ago) 32m >> job-history-6675d8c7b4-xlbdp 1/1 Running 3 (31m ago) 32m >> scheduler-5f4f78b754-9xbrw 1/1 Running 3 (31m ago) 32m >> file-storage-0 1/1 Running 3 (30m ago) 31m >> file-broker-dc7c86d5f-fgg8m 1/1 Running 4 (30m ago) 32m >> type-service-8489f88fb-54r4s 0/1 CrashLoopBackOff 11 (16s ago) 32m >> toolbox-6b9f9b8d64-v6wvl 0/1 CrashLoopBackOff 11 (7s ago) 32m >> ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs >> type-service-8489f88fb-54r4s >> >>> start >>> node src/type-service.js >> npm ERR! code EACCES >> npm ERR! syscall open >> npm ERR! path /home/user/.npm/_cacache/tmp/b7d792dd >> npm ERR! errno -13 >> npm ERR! >> npm ERR! Your cache folder contains root-owned files, due to a bug in >> npm ERR! previous versions of npm which has since been addressed. >> npm ERR! >> npm ERR! To permanently fix this problem, please run: >> npm ERR! sudo chown -R 1000:0 "/home/user/.npm" >> >> npm ERR! Log files were not written due to an error writing to the directory: >> /home/user/.npm/_logs >> npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs >> in your terminal > > I am not able to open a bash inside the type-service, always saying > e.g. error: unable to upgrade connection: container not found ("type-service") > > I tried "type-service-8489f88fb-54r4s", "type-service" and others. Always "not > found". > > Any suggestions? > > Regards, > > Oli > > > > > > > > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech > > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... > https://lists.sourceforge.net/lists/listinfo/chipster-tech -- Oliver Heil Microarray Core Facility Bioinformatics German Cancer Research Center (DKFZ) Foundation under Public Law Im Neuenheimer Feld 580 69120 Heidelberg Germany o....@dk... <mailto:o....@dk...> Support: www.dkfz.de/gpcf/phpBB3/index.php <https://www.dkfz.de/gpcf/phpBB3/index.php> www.dkfz.de <http://www.dkfz.de> Research for a Life without Cancer Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich VAT-ID No.: DE143293537 |
From: <chi...@cs...> - 2022-12-14 06:12:47
|
Hi Oliver, Sorry to hear that there is a problem again with the pods. Unfortunately many people are out of office now (Finland seems to have a wave of covid again), so it will be a little while before we can provide technical help. Best, Eija ----- Original Message ----- From: "Oliver Heil" <o....@dk...> To: "chipster-tech" <chi...@li...> Sent: Tuesday, 13 December, 2022 13:46:00 Subject: [Chipster-tech] pods not coming back again Hi Petri, here I am back again, not like the pods ;-) After going into the update routine just before I planed to upgrade to the newest tool version (which takes probably longer), two of the pods do not come back again. The last time you solved magically by a new container(?) in the repository, so we didn't solved it actually. This is the current state: > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pod > NAME READY STATUS RESTARTS AGE > chipster-session-db-postgresql-0 1/1 Running 1 (63m ago) > 7d22h > chipster-job-history-postgresql-0 1/1 Running 1 (63m ago) > 7d22h > chipster-auth-postgresql-0 1/1 Running 1 (63m ago) > 7d22h > auth-688c6946d-ssxl9 1/1 Running 0 32m > service-locator-787cf87685-97bpq 1/1 Running 2 (31m ago) 32m > session-worker-586f48cc8f-mrnwd 1/1 Running 3 (31m ago) 32m > backup-5bb7dd649f-gbmgj 1/1 Running 3 (31m ago) 32m > web-server-778b9f9f46-6x69x 1/1 Running 3 (31m ago) 32m > session-db-7f49fb79-4nm5z 1/1 Running 3 (31m ago) 32m > job-history-6675d8c7b4-xlbdp 1/1 Running 3 (31m ago) 32m > scheduler-5f4f78b754-9xbrw 1/1 Running 3 (31m ago) 32m > file-storage-0 1/1 Running 3 (30m ago) 31m > file-broker-dc7c86d5f-fgg8m 1/1 Running 4 (30m ago) 32m > type-service-8489f88fb-54r4s 0/1 CrashLoopBackOff 11 (16s ago) 32m > toolbox-6b9f9b8d64-v6wvl 0/1 CrashLoopBackOff 11 (7s ago) 32m > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs > type-service-8489f88fb-54r4s > > > start > > node src/type-service.js > > npm ERR! code EACCES > npm ERR! syscall open > npm ERR! path /home/user/.npm/_cacache/tmp/b7d792dd > npm ERR! errno -13 > npm ERR! > npm ERR! Your cache folder contains root-owned files, due to a bug in > npm ERR! previous versions of npm which has since been addressed. > npm ERR! > npm ERR! To permanently fix this problem, please run: > npm ERR! sudo chown -R 1000:0 "/home/user/.npm" > > npm ERR! Log files were not written due to an error writing to the directory: > /home/user/.npm/_logs > npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs > in your terminal I am not able to open a bash inside the type-service, always saying e.g. error: unable to upgrade connection: container not found ("type-service") I tried "type-service-8489f88fb-54r4s", "type-service" and others. Always "not found". Any suggestions? Regards, Oli _______________________________________________ Chipster-tech mailing list Chi...@li... https://lists.sourceforge.net/lists/listinfo/chipster-tech |
From: Oliver H. <o....@dk...> - 2022-12-13 11:46:20
|
Hi Petri, here I am back again, not like the pods ;-) After going into the update routine just before I planed to upgrade to the newest tool version (which takes probably longer), two of the pods do not come back again. The last time you solved magically by a new container(?) in the repository, so we didn't solved it actually. This is the current state: > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl get pod > NAME READY STATUS RESTARTS AGE > chipster-session-db-postgresql-0 1/1 Running 1 (63m ago) > 7d22h > chipster-job-history-postgresql-0 1/1 Running 1 (63m ago) > 7d22h > chipster-auth-postgresql-0 1/1 Running 1 (63m ago) > 7d22h > auth-688c6946d-ssxl9 1/1 Running 0 32m > service-locator-787cf87685-97bpq 1/1 Running 2 (31m ago) 32m > session-worker-586f48cc8f-mrnwd 1/1 Running 3 (31m ago) 32m > backup-5bb7dd649f-gbmgj 1/1 Running 3 (31m ago) 32m > web-server-778b9f9f46-6x69x 1/1 Running 3 (31m ago) 32m > session-db-7f49fb79-4nm5z 1/1 Running 3 (31m ago) 32m > job-history-6675d8c7b4-xlbdp 1/1 Running 3 (31m ago) 32m > scheduler-5f4f78b754-9xbrw 1/1 Running 3 (31m ago) 32m > file-storage-0 1/1 Running 3 (30m ago) 31m > file-broker-dc7c86d5f-fgg8m 1/1 Running 4 (30m ago) 32m > type-service-8489f88fb-54r4s 0/1 CrashLoopBackOff 11 (16s ago) 32m > toolbox-6b9f9b8d64-v6wvl 0/1 CrashLoopBackOff 11 (7s ago) 32m > ubuntu@chipstervm1:~/git/chipster-openshift/k3s$ kubectl logs > type-service-8489f88fb-54r4s > > > start > > node src/type-service.js > > npm ERR! code EACCES > npm ERR! syscall open > npm ERR! path /home/user/.npm/_cacache/tmp/b7d792dd > npm ERR! errno -13 > npm ERR! > npm ERR! Your cache folder contains root-owned files, due to a bug in > npm ERR! previous versions of npm which has since been addressed. > npm ERR! > npm ERR! To permanently fix this problem, please run: > npm ERR! sudo chown -R 1000:0 "/home/user/.npm" > > npm ERR! Log files were not written due to an error writing to the directory: > /home/user/.npm/_logs > npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs > in your terminal I am not able to open a bash inside the type-service, always saying e.g. error: unable to upgrade connection: container not found ("type-service") I tried "type-service-8489f88fb-54r4s", "type-service" and others. Always "not found". Any suggestions? Regards, Oli |
From: Chipster <chi...@cs...> - 2022-12-08 13:58:06
|
Hi Oli, thanks for the feedback. I added a few words how to restore the backup. Merry Christmas and happy new year! Regards, Petri > On 7. Dec 2022, at 14.26, Oliver Heil <o....@dk...> wrote: > > Hi Petri, > > thanks for the elaboration on this topic, it helps a lot to know, for what > to look and what to try best. > > I will go the road of disallowing LDAP login and allow only a temporary > local account together with temporary changes on the web pages with > the maintenance information somewhere prominent. I already tried it > but the part of invalidation of current logins was missing so far. Well, resetting > all internal keys to new ones seems a bit overkill, but if this is the only > way, than ok. > > You should add some words on how to use the passwords-backup.json file > if something goes wrong. At least I do not have an idea, on how this file > can be used as backup ;-) > > A nginx reverse proxy is something which definitely should be considered, > perhaps even as standard part of the general chipster installation. I just started > to implement it in front of some other apache services, and it makes maintenance > on the apache side so much more comfortable. E.g. you can easily > switch to a second server which provides service during maintenance. The User > does not need to see any difference and there is zero network time overhead, > well, zero is wrong, but you can't feel it. But I don't know anything of the problems > to integrate such a thing into the chipster architecture, so just some motivation > from me to do it :-) > > Regards, > > Oli > > PS: if things are going well now, I wish > > MERRY CHRISTMAS AND A HAPPY NEW YEAR !!! > > to the whole chipster team and enjoy some time off, hopefully > without more bothering support emails from this german dude! > > ;-) > > > > Am 07.12.2022 um 11:46 schrieb Chipster: >> Hi Oli, >> >> that's an interesting use case, which we haven't really thought before. >> >> All external requests to Chipster in K3s come through a proxy called Traefik. Chipster Helm templates configure it with an IngressRoute object: https://github.com/chipster/chipster-openshift/blob/master/k3s/helm/chipster/templates/ingress.yaml <https://github.com/chipster/chipster-openshift/blob/master/k3s/helm/chipster/templates/ingress.yaml> . >> >> It looks like it would be possible to configure Basic auth for the IngressRoute ( https://k3s.rocks/basic-auth/ <https://k3s.rocks/basic-auth/> ), but Chipster Helm templates don't have any support for it at the moment. If it's added manually (for example with command "kubectl edit"), next Chipster deployment will delete it. Authentication would have to be added only to the web-server service ( PathPrefix(`/`) ), which serves the Chipster web app. This would prevent anyone from loading the Chipster app. However, I think you couldn't add authentication to any of the Rest API services like file-broker or session-db, because Chipster don't expect Basic auth there. This means that if anyone would have Chipster open already in their browser, they could continue to use it despite the Basic auth. >> >> Maybe you could use firewall to block all other external IP addresses except your workstation? IngressRoute could use "IP whitelist" ( https://blog.knoldus.com/how-to-whitelist-ips-using-traefik-ingress-controller/ <https://blog.knoldus.com/how-to-whitelist-ips-using-traefik-ingress-controller/> ), but that is not implemented in the Chipster Helm templates either. Maybe easiest could be to configure a firewall in Ubuntu, so that you could add and remove this restriction without touching Chipster configuration. But I have never tried this. >> >> How about if you create a new local user account for yourself and then disable other login methods? Old authentication tokens would be still valid, but I wrote some instructions how you could invalidate those: https://github.com/chipster/chipster-openshift/blob/k3s/k3s/README.md#jwt-keys <https://github.com/chipster/chipster-openshift/blob/k3s/k3s/README.md#jwt-keys> . The downside of this is that the addition and removal of authentication methods requires restarts and users wouldn't know why their account stopped working. >> >> We have been thinking of adding some kind of http proxy (for example Apahce or nginx) in front of our production Chipster that could show a simple info page during service breaks. I don't know yet how easy or difficult this will be, but I'll try to write some instructions about it when we get that far. >> >> Regards, >> Petri >> > > -- > > Oliver Heil > Microarray Core Facility > Bioinformatics > > German Cancer Research Center (DKFZ) > Foundation under Public Law > Im Neuenheimer Feld 580 > 69120 Heidelberg > Germany > > o....@dk... <mailto:o....@dk...> > Support: www.dkfz.de/gpcf/phpBB3/index.php <https://www.dkfz.de/gpcf/phpBB3/index.php> > www.dkfz.de <http://www.dkfz.de/><GxVwsbxrLa9mSRQ0.jpg> > Management Board: Prof. Dr. med. Dr. h. c. Michael Baumann, Ursula Weyrich > VAT-ID No.: DE143293537 > > _______________________________________________ > Chipster-tech mailing list > Chi...@li... <mailto:Chi...@li...> > https://lists.sourceforge.net/lists/listinfo/chipster-tech <https://lists.sourceforge.net/lists/listinfo/chipster-tech> |