Domain Admins
120
edits
Changes
Jump to navigation
Jump to search
Intervention report 2017-01-12 (view source)
Revision as of 17:16, 12 January 2017
, 17:16, 12 January 2017Created page with "# Description Login was broken on all machines of the space and ps1auth was complaining a lot about other stuff. Took over someone that was trying to correct it but had to le..."
# Description
Login was broken on all machines of the space and ps1auth was complaining a lot about other stuff. Took over someone that was trying to correct it but had to leave.
# Intervention by
- bjonnh
# Problem
- Seen some libraries problems on the ps1auth VM. Tried to update them. Then started the clusterfuck.
# Resolution
- Did a full update of arch on ps1auth.
- Only some stuff got through
- Had to disable GPG signing of packages for that using sed to modify pacman.conf, because nothing else was working
- Did the update
- Rebooted bob, resolved the initial problem of machines not logging, but I had to do that two times
- Had to reconfigure network because interface name changed
- As python was updated had to recreate the virtualenv of ps1auth
- Had to update some packages (beautifoulsoup4, kombu add django-ldap something) because they failed with the new python
- Rebooted the machine to be sure everything was ok
- Got home.
# How to avoid the problem in the future
- Well I had to learn how things worked on the fly. We plan on updating the
infrastructure to more maintenable systems (arch was a big problem here as it
was not updated for a long time).
- Have emergency clone VMs to restore quickly (if one has to leave before finishing upgrade/maintenance etc)?
[[Category:Intervention report]][[Category: Systems group]]
Login was broken on all machines of the space and ps1auth was complaining a lot about other stuff. Took over someone that was trying to correct it but had to leave.
# Intervention by
- bjonnh
# Problem
- Seen some libraries problems on the ps1auth VM. Tried to update them. Then started the clusterfuck.
# Resolution
- Did a full update of arch on ps1auth.
- Only some stuff got through
- Had to disable GPG signing of packages for that using sed to modify pacman.conf, because nothing else was working
- Did the update
- Rebooted bob, resolved the initial problem of machines not logging, but I had to do that two times
- Had to reconfigure network because interface name changed
- As python was updated had to recreate the virtualenv of ps1auth
- Had to update some packages (beautifoulsoup4, kombu add django-ldap something) because they failed with the new python
- Rebooted the machine to be sure everything was ok
- Got home.
# How to avoid the problem in the future
- Well I had to learn how things worked on the fly. We plan on updating the
infrastructure to more maintenable systems (arch was a big problem here as it
was not updated for a long time).
- Have emergency clone VMs to restore quickly (if one has to leave before finishing upgrade/maintenance etc)?
[[Category:Intervention report]][[Category: Systems group]]