Effective January 1st 2022, ICE Enterprises will increase all product prices by 20%. Market forces, both rising component cost and increasing material lead time, are requiring the first price adjustment in company history. This price increase will allow ICE Enterprises to continue to design and manufacture quality electronic hardware as has been the case for over two decades. ICE Enterprises will honor all current price quotes and accept orders at existing pricing through the end of December 2021.

ICE-PIC Configuration

From ICE Enterprises
Jump to navigation Jump to search

This page will walk you through setting up your ICE-PIC card on a Red Hat-based architecture. You will need to setup RAM for the PCI-E card. Then you will have to compile and install the ICE-PIC driver.

The ICE-PIC driver is packaged in the ICE Option Tree. The ICE Option Tree is available on our Downloads page.

If you need help setting up the ICE Option Tree, please refer here.

Note: In older versions of the ICE Option Tree users were required to alter the icepic script. This has changed! It is now recommended that you leave the icepic script unaltered. Your specific RamStart, RamDisk and RamMapped settings should now be set on your GRUB line. There are examples of this below.

Prerequisites

Make sure that you have the kernel headers and development files on your machine. The easiest way to install them (and their dependencies) is from a Red Hat repository with yum:

yum install kernel-headers kernel-devel

Memory Settings

We need to get the RAM settings. The RAM Map can be found in the file /var/log/dmesg. Please note that this file has much more than just the RAM Map. We are looking (specifically) for blocks of 'usable' memory.

Here is an example of a RAM Map for one of our ICE Systems:

BIOS-provided physical RAM map:
 BIOS-e820: 0000000000010000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 0000000020000000 (usable)
 BIOS-e820: 0000000020000000 - 0000000020200000 (reserved)
 BIOS-e820: 0000000020200000 - 0000000040000000 (usable)
 BIOS-e820: 0000000040000000 - 0000000040200000 (reserved)
 BIOS-e820: 0000000040200000 - 00000000be9cd000 (usable)
 BIOS-e820: 00000000be9cd000 - 00000000bea1a000 (ACPI NVS)
 BIOS-e820: 00000000bea1a000 - 00000000bea31000 (reserved)
 BIOS-e820: 00000000bea31000 - 00000000bea34000 (ACPI NVS)
 BIOS-e820: 00000000bea34000 - 00000000bea37000 (reserved)
 BIOS-e820: 00000000bea37000 - 00000000bea45000 (ACPI data)
 BIOS-e820: 00000000bea45000 - 00000000bea89000 (reserved)
 BIOS-e820: 00000000bea89000 - 00000000bea8a000 (usable)
 BIOS-e820: 00000000bea8a000 - 00000000bea92000 (reserved)
 BIOS-e820: 00000000bea92000 - 00000000bea93000 (ACPI NVS)
 BIOS-e820: 00000000bea93000 - 00000000bea97000 (reserved)
 BIOS-e820: 00000000bea97000 - 00000000beab0000 (ACPI NVS)
 BIOS-e820: 00000000beab0000 - 00000000bead9000 (reserved)
 BIOS-e820: 00000000bead9000 - 00000000beb1c000 (ACPI NVS)
 BIOS-e820: 00000000beb1c000 - 00000000bedbb000 (usable)
 BIOS-e820: 00000000bedbb000 - 00000000beff7000 (reserved)
 BIOS-e820: 00000000beff7000 - 00000000bf000000 (usable)
 BIOS-e820: 00000000bf800000 - 00000000cfa00000 (reserved)
 BIOS-e820: 00000000f8000000 - 00000000fc000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fed00000 - 00000000fed04000 (reserved)
 BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 000000042fe80000 (usable)

Determining RAM Information

You can see from the RAM map that the largest block of usable RAM is the last line:

BIOS-e820: 0000000100000000 - 000000042fe80000 (usable)

Using this block, we can calculate an allowable start address and our Top of Memory (ToM). The start address and ToM must be an integer. Therefore, we'll convert 0x100000000 and 0x42fe00000 to bytes, divide by 1048576 (1024*1024) then take just the whole number.

Start Address

0x0000000100000000 = 4294967296 (4096M)

Top of Memory (ToM)

0x000000042fe80000 = 17983602688 (17150M)

We'll now show you two examples of valid ICE-PIC memory configurations using these two values (4096M and 17150M).

Example 1

Let's maximize our RamDisk Size. This is done by starting our RamDisk at the first available address (4096M) and mapping only 24M. By doing this, the Linux kernel will be restricted to 4096M.

RamStart = 4096M
RamMapped = 24M
RamDisk = ToM - ( RamStart + RamMapped ) 
-> RamDisk = 17150M - ( 4096M + 24M )
RamDisk = 13030M

Example 2

In this example we're going to give the Linux kernel more memory. We'll increase kernel's memory restriction from 4096M to 8192M.

RamStart = 8192M
RamMapped = 24M
RamDisk = ToM - ( RamStart + RamMapped ) 
-> RamDisk = 17150M - ( 8192M + 24M )
RamDisk = 8934M

Setting the Kernel Line

In order to setup the ICE-PIC, we need to change some boot parameters. We must restrict the kernel's memory and set the properties for our RamDisk. This is done by adding the correct parameters to the GRUB boot line located in /boot/grub/grub.conf.

We're going to show you two examples of GRUB configuration entries. The first entry is for one of our Red Hat 5 machines. The second entry is for one of our Red Hat 6 machines. Both examples will use the following settings:

  • RamStart = 4096M
  • RamDisk = 13030M
  • RamMapped = 24M

Note: Your GRUB file may have multiple entries. Also, grub.conf uses zero-based indexing (i.e., default=1 means that the 2nd entry will be used)

RHEL/CentOS 5

This is a GRUB configuration for one of our CentOS 5.9 machines:

# grub.conf generated by anaconda
#
default=0
timeout=3
#
title ICE-MACHINE
       root (hd0,0)
       kernel /boot/vmlinuz-2.6.18-308.1.1.el5 ro root=LABEL=ICESYS selinux=0 video=vesafb:nomtrr vga=773 elevator=deadline mem=4096M ice.rs=4096M ice.rd=13030M ice.rm=24M
       initrd /boot/initrd-2.6.18-308.1.1.el5-MACHINE.img
  • mem=4096M restricts the Linux kernel to 4096M
  • ice.rs=4096M sets the RamDisk starting address to 4096M
  • ice.rd=13030M sets the RamDisk size to 13030M
  • ice.rm=24M sets the RamDisk Mapped Memory to 24M

RHEL/CentOS 6

This is a GRUB configuration for one of our Red Hat 6.4 machines:

# grub.conf generated by anaconda
#
default=0
timeout=3
#
title ICE-MACHINE
       root (hd0,0)
       kernel /vmlinuz-2.6.32-358.6.2.el6.x86_64 ro root=LABEL=ICESYS rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM nomodeset vga=791 elevator=deadline memmap=13054M$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M processor.max_cstate=0 intel_idle.max_cstate=0
       initrd /initramfs-2.6.32-358.6.2.el6.x86_64.img

Note: RamDisk(13030M) + RamMapped(24M) = 13054M

  • memmap=13054M$4096M reserves 13054M starting @ address 4096M
  • ice.rs=4096M sets the RamDisk starting address to 4096M
  • ice.rd=13030M sets the RamDisk size to 13030M
  • ice.rm=24M sets the RamDisk Mapped Memory to 24M

RHEL/CentOS 7

RHEL 7 provides multiple ways of configuring the GRUB2 bootloader: append the kernel line or create a custom menuentry. Note that you must recreate the grub.cfg file (as root) and reboot for any changes to take effect:

grub2-mkconfig -o /boot/grub2/grub.cfg
...
reboot

We recommend backing up the existing grub.cfg before proceeding.

Option 1: GRUB Default File

Edit /etc/default/grub appending the GRUB_CMDLINE_LINUX line as shown below:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto elevator=deadline idle=poll processor.max_cstate=0 intel_idle.max_cstate=1 intel_pstate=disable memmap=13054M\\\$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M"
GRUB_DISABLE_RECOVERY="true"
Please note that the '$' in the memmap option must be escaped with '\' under RHEL 7. Additionally, in /etc/default/grub, the escape characters themselves must be escaped. Thus, the memmap value must have three (3) blackslashes to escape the '$'. After running grub2-mkconfig this resolves to the correct value: memmap=13054M\$4096M

Configuration Details:

RamDisk(13030M) + RamMapped(24M) = 13054M

  • memmap=13054M\$4096M reserves 13054M starting @ address 4096M
  • ice.rs=4096M sets the RamDisk starting address to 4096M
  • ice.rd=13030M sets the RamDisk size to 13030M
  • ice.rm=24M sets the RamDisk Mapped Memory to 24M

Option 2: Custom Menu Entry

A more flexible method of configure the kernel line parameters is to create a custom menu entry. The main advantage to using a custom menu entry for our purposes is that it allows the original menu entry to remain intact if there is a significant error in the new custom default entry which renders the machine unbootable. In this case, the machine will still boot by selecting the original menu entry at the boot menu.

While there are various options, we recommend using the RHEL 7 supplied configuration file /etc/grub.d/40_custom. Do not delete the header of the file, but rather add one or more menuentry sets based on the following form:

menuentry 'ICE-MACHINE' ... {
        ...
        set root='hd0,msdos1'
        linux16 /vmlinuz-3.10.0-693.el7.x86_64 ro root=LABEL=ICESYS ro crashkernel=auto rd.luks=0 nomodeset modeset=0 elevator=deadline idle=poll processor.max_cstate=0 intel_idle.max_cstate=1 intel_pstate=disable memmap=13054M\$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M
        initrd16 /initramfs-3.10.0-693.el7.img
}
Please note that the '$' in the memmap option must be escaped with '\' under RHEL 7.

Manual vs. Kernel Allocated Memory

The above examples detail how to add kernel options which manually specify the size of the icedisk memory buffer and the location in memory at which it starts with memmap; however, in some cases, higher performance can be achieved by using kernel allocated memory to enable direct I/O. This is done by means of the ice.ra option on the kernel line with a value of 2. In this case, the memmap and ice.rs options are not needed. Using the values from the previous examples, the kernel allocated memory configuration can be done by appending the kernel line with the following:

... elevator=deadline idle=poll processor.max_cstate=0 intel_idle.max_cstate=1 intel_pstate=disable ice.ra=2 ice.rd=13030M ice.rm=24M

Reboot

A reboot is required after altering the grub.conf file. After the reboot, you can verify your settings at a Linux terminal by typing:

$ cat /proc/cmdline

Your output should look something like this:

ro root=LABEL=ICESYS rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 crashkernel=131M@0M rd_NO_LVM rd_NO_DM nomodeset vga=791 elevator=deadline memmap=13054M$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M processor.max_cstate=0 intel_idle.max_cstate=0

where memmap, ice.rs, ice.rd and ice.rm equal the values in your grub.conf file.

ICE-PIC Driver

Once all the RAM settings are configured, we can compile the ICE-PIC driver. This script is located in the ICE Option Tree (which is assumed to be '/opt/ice/pkg/icexxx'). Start by becoming the root user and entering into the correct directory:

$ su -l
(enter password) 
$ cd /opt/ice/pkg/icexxx/drv/lnx

If you already have an ICE-PIC driver installed, you'll need to stop and remove it. (If you aren't sure, you can always try to run cat /proc/driver/icepic.)

$ ./icepic stop
$ ./icepic remove

Now make the driver:

$ ./icepic make

Make sure that there are no errors in the terminal.

If you didn't properly install the Linux Headers & Development files, then you will see the following error:

make: *** /lib/modules/W.X.Y-Z/build: No such file of directory. Stop.

where W.X.Y-Z is your Linux kernel version. Please refer back to the prerequisite section for resolution.

If you were able to run the make without any errors, then it is safe to install and start the driver:

$ ./icepic install
$ ./icepic start

Validation

To confirm that your ICE-PIC card is properly configured, type

$ cat /proc/driver/icepic

Your output should match your settings and look something like this:

RamStart  : 4096 Mby
RamDisk   : 13030 Mby
RamMapped : 24 Mby
Ram Addr  : Start=0x100000000 End=0x42fe00000
Buffers   : 1
Buffer #0 : Addr=0x100000000 Size=0x32e600000 
Devices   : 1
Device #0 : Revision=0x67 MemBase=0xf7b00000   Card Temp=39 MJS=1370027723   Peak Temp=39 MJS=1369948462   Comp Temp=(39,-1,-1,-1,-1)
Version   : 361