OpenVMS Notes: x86-64 Porting Diaries

edit: 2025-01-31

Bare-metal install of OpenVMS-9.2-2 x86-64 on HPE DL380 Gen9

• OpenVMS-9.2-2 x86-64 can only be installed on hardware that employs UEFI. BIOS-only machines will not work.

  Machine	Console Firmware	OpenVMS Possible?	Notes
  HP DL380p Gen8	BIOS	N	380p = performance; 380e = entry model
  HPE DL380 Gen9	UEFI	Y

• The official 922 installation docs found at the VSI site are really well written so you might not need the following information (although they seem to focus a little more on installing OpenVMS on a Virtual Machine). What follows are some problems I encountered when doing a bare-metal install on an HPE DL380 Gen9
• OpenVMS installs on any system technology will require you to communicate with OpenVMS via terminal port OPA0, and the x86 install is no different. The first steps are GUI-based which require a VGA monitor to run UEFI program VMS_BOOTMGR which usually defaults to device COM1 (which becomes OPA0).
  • OpenVMS-9.2-3 CAVEAT: the updated VMS_BOOTMGR introduces a new boot flag indicator called OPA0, and commands: OPA, OPA0, and NOOPA. So if you are installing or upgrading to OpenV-MS-9.2-3 then first attempt to do everything from the VGA console without messing around with COM1 or the iLO port.
• Since we will eventually need to connect to OPA0 via COM1, I connected a via the iLO then played with the VSP (Virtual Serial Port) command which only connects to COM2
• The VSI docs indicate that the COM command inside VMS_BOOTMGR will be able to connect to COM2 but this was all that was available on my system.

  command	meaning
  COM	display current setting
  COM 1	enable COM 1
  COM 0	disable COM 1

  Note that COM1 was already enabled but was not working for me. There was no VMS_BOOTMGR command to change to device COM 2 so I needed to modify a BIOS setting of this machine so that the VSP (virtual serial port) command connects to COM1
  [[[ What? I just finished saying that the HPE DL380 Gen9 supports UEFI so how do you change a BIOS setting? I suggest you poke around your console firmware looking for the Legacy BIOS menu (I left out the actual command because it can change dependent upon your local firmware version) ]]]

Prepping the media

• Many servers (and personal computers) today do not host any optical disk drives so I use a USB-based ASUS (SDRW-08D2S-U) Slim External 8x DVD Writer
  • CAVEAT: this beast is power hungry so both USB connectors (on the drive) must be plugged in for both reading as well as writing.
  • The HPE DL380 Gen9 did not have enough USB ports so I was forced to use a so-called USB power bar. Twenty bucks on Amazon.
• I logged into the VSI portal then downloaded the kit which included file X860922.ISO which is 1.44 GB in size making easy for writing to a single layer DVD.
  • Because the ISO file represents a multi-partition image, you must use burner software that will preserve all partitions.
  • I used this: Express Burn by NCH Software. Select "tab: ISO" then click on "Write ISO"

using the iLO (rather than the serial port)

• Connecting to the iLO (from another OpenVMS system via the MULTINET stack)

• KAWC90(newPROD)::Neil> ssh 192.168.255.77
  Host key not found from database.
  Key fingerprint(ssh-dss):
  ssh-dss sha1: xugis-dukim-fybyt-myhod-mesuh-deciz-sigoz-rikan-vavok-desed-tixux
  ssh-dss sha256: xohet-nypan-havyc-setam-tezuv-sugyg-vyvec-haduh-nyrip-zazef-temeg-sekit-dabyb-kofim-lorel-vacul-nyxix
  You can get a public key's fingerprint by running
  (OpenVMS) $ MULTINET sshkeygen /ssh2/fingerprint=publickey.pub
  (UNIX)    % ssh-keygen -F publickey.pub
  on the keyfile.
  Are you sure you want to continue connecting (yes/no)? yes
  Host key saved to CSMIS$USER3:[ADMCSM.NEIL.SSH2.HOSTKEYS]key_22_192_168_255_77.pub
  host key for 192.168.255.77, accepted by neil Fri Feb 07 2025 07:39:16 -0500
  
  neil's password: 
  
  Authentication successful.
  User:neil logged-in to kawc7777.on.bell.ca(192.168.255.77 / FE80::32E1:71FF:FE65:DC64)
  
  iLO Advanced 2.82 at  Feb 06 2023
  Server Name: 
  Server Power: On
  
  </>hpiLO-> vsp
  
  Virtual Serial Port Active: COM1
  
  Starting virtual serial port.
  Press 'ESC (' to return to the CLI Session.

• Connecting to the iLO (from a Linux system)
  
  • I was unable to use SSH (from a console session) in AlmaLinux-8 to fully connect to the iLO
    (I was able to connect, but my client and iLO server could not find common ground on the initial handshake)
  • The reason? Starting with EL-8 (Enterprise Linux 8 which includes: AlmaLinux, CentOS, EuroLinux, RHEL, Rocky Linux, Oracle Linux, etc.), the system-wide libraries have been pruned of encryption algorithms with known exploits. It gets worse with EL9 (RHEL-9 system cannot communicate with RHEL-6 systems without a lot of fiddling around)
  • This affects native Linux programs like SSH and SSHd so you need to use an application that maintains both the old and new algorithms. Fortunately, a Linux version of PuTTY (has its own private library of encryption algorithms) was available and this worked like a charm:
    

    $ yum install epel-release	# add the repo: Extra Packages for Enterprise Linux
    $ yum install putty		# add putty

  • You may now connect to the iLO (the server needs to be connected to power, but not necessarily powered on)
    

    $ putty -ssh 192.168.255.77

  • From this point you only need to log in (the account name and password will be found on a pull-out tag at the front of the server
  • Type HELP
  • Now type VSP (virtual serial port).
    • If it connects to COM 1 then all is well so hit the <ESC> key followed by the "(" key to exit VSP mode for now
    • If it connects to COM 2 then your firmware needs to be reconfigured so it will connect to COM 1so power up the server
      OVERVIEW:
    • unplug the USB-Drive (see reason below) then power-up the server.
    • Hit the F9 key as soon as you see the F9-F10-F11 prompts on the VGA monitor
    • Navigate these menu items:
      • System Configuration
      • BIOS/Platform Configuration (RBSU)
      • System Options
      • Serial Port Options
        • Change "Virtual Serial Port" from COM2 to COM1
        • optional: Change "Embedded Serial Port" to DISABLED
        • hit the <F10> key to save the settings.
        • now back out of all these menus until the server reboots

Steps

1. unplug the USB-DVD
   • Why? power-on diagnostics have been known to clobber writable USB media. Oh why didn't those early USB developers include a write-protect switch?
2. power up the server (again)
3. wait until you see the prompt for keys: F9, F10, F11
   • hit the F10 key:
     •  if you wish to setup your RAID controller and/or iLO port graphically
       
       • Note that one drive (or RAID set) needs to be designated DEFAULT in order for it to be exposed to console firmware.
   • hit the F9 key:
     • if you wish to setup your RAID controller and/or iLO port non-graphically
       • Note that one drive (or RAID set) needs to be designated DEFAULT in order for it to be exposed to console firmware.
     • to run any console programs including UEFI Shell (which you will use to find/start VMS_BOOTMGR)

Legend:
  <ur>       = user response
  <sr>       = system response
  <enter>    = hit the "Enter Key"
  [[ View ]] = some higher level activity
===============================================================
<ur>  [[ Physical Action ]] Plug in the USB-DVD then wait for the drive LEDs to stop blinking
<ur>  [[ GUI Action ]] Select "UEFI Shell"
<sr>  Shell>
<ur>  map -r
      ("-r" forces the map command to rescan all the hardware)
<sr>  [[ displays a list of block structured devices (BLK0, etc) ]]
         1 If any of these contain a file system, then you will see a second entry (FS0, etc)
         2 FS0 should point to your DVD (if this is your first time)
         3 If you do not see any FS devices then your DVD burner failed to preserve the EFI partition
         4 On a VMS update, the DVD drive will most likely come up as FS1
<ur>  fs0:<enter>
<sr>  fs0:\>
<ur>  dir
<sr>  [[ View ]] directory listing (efi must be one of them; this is where the UEFI/EFI files will be found)
<ur>  cd efi
<sr>  fs0:\efi\
<ur>  dir
<ur>  [[ View ]] directory listing
<sr>  cd vms
<sr>  fs0:\efi\vms\
<ur>  [[ View ]] directory listing
<sr>  vms_bootmgr
<sr>  View: GUI-based tool
=========================================================================================
<ur>  using the iLO port, connect to Virtual Serial Port via the "VSP"command
=========================================================================================
<ur>  dev
      (displays available media)
<sr>  dna5020
      (this is my optical drive)
<ur>  boot dna5020
=========================================================================================
<sr>  Graphics i/o is now suspended. Installation continues on COM1 (via "VSP")
=========================================================================================
<sr>  VMS installation menu is presented (choose "item 1" for INSTALL)

Part-2

If everything works properly over then next few reboots, then use ...
... the VMS_BOOTMGR command "AUTO" to configure your system for an automatic boot.
Note: This is different from what you might do on an Itanium

---

 Back to Home
Neil Rieck
Waterloo, Ontario, Canada.