ICE-PIC Configuration
This page will walk you through setting up your ICE-PIC card on a Red Hat-based architecture. You will need to setup RAM for the PCI-E card. Then you will have to compile and install the ICE-PIC driver.
The ICE-PIC driver is packaged in the ICE Option Tree. The ICE Option Tree is available on our Downloads page.
If you need help setting up the ICE Option Tree, please refer here.
Note: In older versions of the ICE Option Tree users were required to alter the icepic script. This has changed! It is now recommended that you leave the icepic script unaltered. Your specific RamStart, RamDisk and RamMapped settings should now be set on your GRUB line. There are examples of this below.
Contents
Prerequisites
Make sure that you have the kernel headers and development files on your machine. The easiest way to install them (and their dependencies) is from a Red Hat repository with yum:
yum install kernel-headers kernel-devel
Memory Settings
We need to get the RAM settings. The RAM Map can be found in the file /var/log/dmesg. Please note that this file has much more than just the RAM Map. We are looking (specifically) for blocks of 'usable' memory.
Here is an example of a RAM Map for one of our ICE Systems:
BIOS-provided physical RAM map: BIOS-e820: 0000000000010000 - 000000000009fc00 (usable) BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved) BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 0000000020000000 (usable) BIOS-e820: 0000000020000000 - 0000000020200000 (reserved) BIOS-e820: 0000000020200000 - 0000000040000000 (usable) BIOS-e820: 0000000040000000 - 0000000040200000 (reserved) BIOS-e820: 0000000040200000 - 00000000be9cd000 (usable) BIOS-e820: 00000000be9cd000 - 00000000bea1a000 (ACPI NVS) BIOS-e820: 00000000bea1a000 - 00000000bea31000 (reserved) BIOS-e820: 00000000bea31000 - 00000000bea34000 (ACPI NVS) BIOS-e820: 00000000bea34000 - 00000000bea37000 (reserved) BIOS-e820: 00000000bea37000 - 00000000bea45000 (ACPI data) BIOS-e820: 00000000bea45000 - 00000000bea89000 (reserved) BIOS-e820: 00000000bea89000 - 00000000bea8a000 (usable) BIOS-e820: 00000000bea8a000 - 00000000bea92000 (reserved) BIOS-e820: 00000000bea92000 - 00000000bea93000 (ACPI NVS) BIOS-e820: 00000000bea93000 - 00000000bea97000 (reserved) BIOS-e820: 00000000bea97000 - 00000000beab0000 (ACPI NVS) BIOS-e820: 00000000beab0000 - 00000000bead9000 (reserved) BIOS-e820: 00000000bead9000 - 00000000beb1c000 (ACPI NVS) BIOS-e820: 00000000beb1c000 - 00000000bedbb000 (usable) BIOS-e820: 00000000bedbb000 - 00000000beff7000 (reserved) BIOS-e820: 00000000beff7000 - 00000000bf000000 (usable) BIOS-e820: 00000000bf800000 - 00000000cfa00000 (reserved) BIOS-e820: 00000000f8000000 - 00000000fc000000 (reserved) BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) BIOS-e820: 00000000fed00000 - 00000000fed04000 (reserved) BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved) BIOS-e820: 0000000100000000 - 000000042fe80000 (usable)
Determining RAM Information
You can see from the RAM map that the largest block of usable RAM is the last line:
BIOS-e820: 0000000100000000 - 000000042fe80000 (usable)
Using this block, we can calculate an allowable start address and our Top of Memory (ToM). The start address and ToM must be an integer. Therefore, we'll convert 0x100000000 and 0x42fe00000 to bytes, divide by 1048576 (1024*1024) then take just the whole number.
Start Address
0x0000000100000000 = 4294967296 (4096M)
Top of Memory (ToM)
0x000000042fe80000 = 17983602688 (17150M)
We'll now show you two examples of valid ICE-PIC memory configurations using these two values (4096M and 17150M).
Example 1
Let's maximize our RamDisk Size. This is done by starting our RamDisk at the first available address (4096M) and mapping only 24M. By doing this, the Linux kernel will be restricted to 4096M.
RamStart = 4096M RamMapped = 24M RamDisk = ToM - ( RamStart + RamMapped ) -> RamDisk = 17150M - ( 4096M + 24M ) RamDisk = 13030M
Example 2
In this example we're going to give the Linux kernel more memory. We'll increase kernel's memory restriction from 4096M to 8192M.
RamStart = 8192M RamMapped = 24M RamDisk = ToM - ( RamStart + RamMapped ) -> RamDisk = 17150M - ( 8192M + 24M ) RamDisk = 8934M
Setting the Kernel Line
In order to setup the ICE-PIC, we need to change some boot parameters. We must restrict the kernel's memory and set the properties for our RamDisk. This is done by adding the correct parameters to the GRUB boot line located in /boot/grub/grub.conf.
We're going to show you two examples of GRUB configuration entries. The first entry is for one of our Red Hat 5 machines. The second entry is for one of our Red Hat 6 machines. Both examples will use the following settings:
- RamStart = 4096M
- RamDisk = 13030M
- RamMapped = 24M
Note: Your GRUB file may have multiple entries. Also, grub.conf uses zero-based indexing (i.e., default=1 means that the 2nd entry will be used)
RHEL/CentOS 5
This is a GRUB configuration for one of our CentOS 5.9 machines:
# grub.conf generated by anaconda
#
default=0
timeout=3
#
title ICE-MACHINE
root (hd0,0)
kernel /boot/vmlinuz-2.6.18-308.1.1.el5 ro root=LABEL=ICESYS selinux=0 video=vesafb:nomtrr vga=773 elevator=deadline mem=4096M ice.rs=4096M ice.rd=13030M ice.rm=24M
initrd /boot/initrd-2.6.18-308.1.1.el5-MACHINE.img
- mem=4096M restricts the Linux kernel to 4096M
- ice.rs=4096M sets the RamDisk starting address to 4096M
- ice.rd=13030M sets the RamDisk size to 13030M
- ice.rm=24M sets the RamDisk Mapped Memory to 24M
RHEL/CentOS 6
This is a GRUB configuration for one of our Red Hat 6.4 machines:
# grub.conf generated by anaconda
#
default=0
timeout=3
#
title ICE-MACHINE
root (hd0,0)
kernel /vmlinuz-2.6.32-358.6.2.el6.x86_64 ro root=LABEL=ICESYS rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM nomodeset vga=791 elevator=deadline memmap=13054M$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M processor.max_cstate=0 intel_idle.max_cstate=0
initrd /initramfs-2.6.32-358.6.2.el6.x86_64.img
Note: RamDisk(13030M) + RamMapped(24M) = 13054M
- memmap=13054M$4096M reserves 13054M starting @ address 4096M
- ice.rs=4096M sets the RamDisk starting address to 4096M
- ice.rd=13030M sets the RamDisk size to 13030M
- ice.rm=24M sets the RamDisk Mapped Memory to 24M
RHEL/CentOS 7
RHEL 7 provides multiple ways of configuring the GRUB2 bootloader: append the kernel line or create a custom menuentry. Note that you must recreate the grub.cfg file (as root) and reboot for any changes to take effect:
grub2-mkconfig -o /boot/grub2/grub.cfg ... reboot
We recommend backing up the existing grub.cfg before proceeding.
Option 1: GRUB Default File
Edit /etc/default/grub appending the GRUB_CMDLINE_LINUX line as shown below:
GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=auto elevator=deadline idle=poll processor.max_cstate=0 intel_idle.max_cstate=1 intel_pstate=disable memmap=13054M\\\$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M" GRUB_DISABLE_RECOVERY="true"
Configuration Details:
RamDisk(13030M) + RamMapped(24M) = 13054M
- memmap=13054M\$4096M reserves 13054M starting @ address 4096M
- ice.rs=4096M sets the RamDisk starting address to 4096M
- ice.rd=13030M sets the RamDisk size to 13030M
- ice.rm=24M sets the RamDisk Mapped Memory to 24M
Option 2: Custom Menu Entry
A more flexible method of configure the kernel line parameters is to create a custom menu entry. The main advantage to using a custom menu entry for our purposes is that it allows the original menu entry to remain intact if there is a significant error in the new custom default entry which renders the machine unbootable. In this case, the machine will still boot by selecting the original menu entry at the boot menu.
While there are various options, we recommend using the RHEL 7 supplied configuration file /etc/grub.d/40_custom. Do not delete the header of the file, but rather add one or more menuentry sets based on the following form:
menuentry 'ICE-MACHINE' ... {
...
set root='hd0,msdos1'
linux16 /vmlinuz-3.10.0-693.el7.x86_64 ro root=LABEL=ICESYS ro crashkernel=auto rd.luks=0 nomodeset modeset=0 elevator=deadline idle=poll processor.max_cstate=0 intel_idle.max_cstate=1 intel_pstate=disable memmap=13054M\$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M
initrd16 /initramfs-3.10.0-693.el7.img
}
Manual vs. Kernel Allocated Memory
The above examples detail how to add kernel options which manually specify the size of the icedisk memory buffer and the location in memory at which it starts with memmap; however, in some cases, higher performance can be achieved by using kernel allocated memory to enable direct I/O. This is done by means of the ice.ra option on the kernel line with a value of 2. In this case, the memmap and ice.rs options are not needed. Using the values from the previous examples, the kernel allocated memory configuration can be done by appending the kernel line with the following:
... elevator=deadline idle=poll processor.max_cstate=0 intel_idle.max_cstate=1 intel_pstate=disable ice.ra=2 ice.rd=13030M ice.rm=24M
Reboot
A reboot is required after altering the grub.conf file. After the reboot, you can verify your settings at a Linux terminal by typing:
$ cat /proc/cmdline
Your output should look something like this:
ro root=LABEL=ICESYS rd_NO_LUKS KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 crashkernel=131M@0M rd_NO_LVM rd_NO_DM nomodeset vga=791 elevator=deadline memmap=13054M$4096M ice.rs=4096M ice.rd=13030M ice.rm=24M processor.max_cstate=0 intel_idle.max_cstate=0
where memmap, ice.rs, ice.rd and ice.rm equal the values in your grub.conf file.
ICE-PIC Driver
Once all the RAM settings are configured, we can compile the ICE-PIC driver. This script is located in the ICE Option Tree (which is assumed to be '/opt/ice/pkg/icexxx'). Start by becoming the root user and entering into the correct directory:
$ su -l (enter password) $ cd /opt/ice/pkg/icexxx/drv/lnx
If you already have an ICE-PIC driver installed, you'll need to stop and remove it. (If you aren't sure, you can always try to run cat /proc/driver/icepic.)
$ ./icepic stop $ ./icepic remove
Now make the driver:
$ ./icepic make
Make sure that there are no errors in the terminal.
If you didn't properly install the Linux Headers & Development files, then you will see the following error:
make: *** /lib/modules/W.X.Y-Z/build: No such file of directory. Stop.
where W.X.Y-Z is your Linux kernel version. Please refer back to the prerequisite section for resolution.
If you were able to run the make without any errors, then it is safe to install and start the driver:
$ ./icepic install $ ./icepic start
Validation
To confirm that your ICE-PIC card is properly configured, type
$ cat /proc/driver/icepic
Your output should match your settings and look something like this:
RamStart : 4096 Mby RamDisk : 13030 Mby RamMapped : 24 Mby Ram Addr : Start=0x100000000 End=0x42fe00000 Buffers : 1 Buffer #0 : Addr=0x100000000 Size=0x32e600000 Devices : 1 Device #0 : Revision=0x67 MemBase=0xf7b00000 Card Temp=39 MJS=1370027723 Peak Temp=39 MJS=1369948462 Comp Temp=(39,-1,-1,-1,-1) Version : 361
