OpenCL on Ubuntu 10.04 (Lucid) with Two ATI FirePro V7800 GPUs

We recently received donated GPUs from ATI and setup two of them within a single Dell T7500 Workstation. Our OS of choice is Ubuntu Lucid 10.04. By default the OS can utilize a single GPU using the Xorg-ati driver. However, without the 3rd party ATI driver (from http://support.amd.com/us/gpudownload/Pages/index.aspx) our system is incapable of running OpenGL with direct rendering and/or using the second GPU.


The first step to install the ATI driver was to get the appropriate installer for our hardware. 
ATI is not known for creating a single installer that has support for ALL of their GPUs. Current Catalyst driver versions available today are 10.11, 10.10, 10.9 (*), 10.8, 10.7 (*), etc. The FirePRO V7800 hardware is intended for professional workstations and can be considered on par with NVidia Quadro GPUs. Because of this, the FirePROs are assigned new drivers that are considered more stable than the ATI Radeon drivers. I took the time to try driver versions 10.11 and 10.10 on our system, but after installation, "aticonfig" reported that no supported hardware was found.
ATI's webpage for Stream SDK 2.2 (supporting OpenCL 1.1) says that Catalyst 10.7 was verified to function with FirePRO V7800s. However, on ATIs driver search page specifies Catalyst 10.9 (Driver v8.773) for the V7800s. I starred both versions in the list above. 
I started with the 10.7 driver. Installation failed on Ubuntu because of a DKMS problem (discussed here: http://ubuntu-ky.ubuntuforums.org/showthread.php?t=1584684). The same problem was encountered with the 10.9 driver provided by the ATI driver download page. My third attempt was to use the hotfixed 10.9 driver that was linked to in the Ubuntu forum)---that attempt also failed because the driver did not have support for our hardware built into it. My original thought was that Catalyst 10.9 was the same no matter how you look at it, but its not the case. 
It turns out after all this headache that I did find a way to get the ATI driver installed. First I downloaded the Catalyst 10.9 driver (8.773). And followed similar tricks to those listed here: http://ubuntuforums.org/archive/index.php/t-1615594.html. (NOTE: I put any patches required in http://sc.fsu.edu/~efb06/vislab/patches)

These commands build Catalyst 10.9 for Ubuntu Lucid 10.04 and our FirePRO V7800 GPUs: 

1. ./ati-driver-installer-10-9-x86.x86_64.run --extract ./extracted
2. cd extracted
3. wget http://sc.fsu.edu/~efb06/vislab/patches/fglrx-2.6.36.patch
4. patch -p1 < fglrx-2.6.36.patch
5. wget http://sc.fsu.edu/~efb06/vislab/patches/fglrx-compat_alloc.patch
6. patch -p1 < fglrx-compat_alloc.patch
7. wget http://sc.fsu.edu/~efb06/vislab/patches/makefile_compat.patch
8. patch -p1 < makefile_compat.patch 
9. ./ati-installer.sh 8.773 --buildpkg Ubuntu/lucid

Those commands build 4 *.deb bundles in the parent directory (for me ~efb06/Downloads/fglrx-8.773). Probably the most interesting thing here is that these *.deb bundles will work without rebuilding in our Ubuntu setup until we upgrade beyond Ubuntu Lucid (10.04). Even when we go beyond Lucid there is a chance they will continue to work, but by then we should have new ATI drivers which will be easier to install (right ATI?). 

With the driver patched and built, installation and setup is done as follows: 

1. sudo dpkg -i ~efb06/Downloads/fglrx-8.773/*.deb
2. sudo service gdm stop
3. sudo aticonfig --initial --adapter=all -f
4. sudo aticonfig --resolution=0,2560x1600 --resolution=1,640x480
5. sudo apt-get install kdm
6. sudo service kdm stop
7. sudo service gdm stop
8. wget http://sc.fsu.edu/~efb06/vislab/patches/kdmrc_noXsecurity.patch
9. sudo patch -p0 < kdmrc_noXsecurity.patch
10. cd /
11. sudo tar xvfz ~efb06/Downloads/icd-registration.tgz
12. sudo service kdm start
13. export DISPLAY=${DISPLAY%.*} (NOTE: I found COMPUTE is a better way to do this)
14. export COMPUTE=:0
15. fglrxinfo
    

A few notes:

1. command #3 and #4 initialize our xorg.conf for separate X sessions on all GPUs. ATI requires all GPUs running OpenCL to have an active X session running, but no size is mandatory. The larger the resolution, the more memory and processing power is required. Since we only have one monitor for the workstation and want a second GPU only for computing, we minimize the resolution and maximize the free memory. 
2. command #5 is required because GDM does not allow us to disable access control on the Xorg server which means we cannot access ATI GPUs via SSH unless we are root. This command in combination with the patch in #8 and #9 where we allow general access to the GPUs via SSH.
3. command #9 reconfigures KDM to launch all X servers with argument "-ac" which effectively allows anyone to access the GPUs and put anything they want on the display. This is insecure, but necessary for remote GPU computing. 
4. command #13-14 is required to make all GPUs visible to ATI's implementation of OpenCL. It turns out that ATI will only utilize GPUs selected by the DISPLAY variable. If DISPLAY=:0.0 then only the first GPU will be used. DISPLAY=:0.1 selects only the second GPU, but DISPLAY=:0 selects both GPUs. (More details can be found here: http://developer.amd.com/gpu_assets/App_Note-Running_ATI_Stream_Apps_Remotely.pdf).  (***UPDATE: I made a change above after I found out that ATI OpenCL also chooses the target hardware based on the COMPUTE variable. Same selection rules as DISPLAY apply, but COMPUTE overrides DISPLAY. This means we can compute remotely without requiring X forwarding and also without accidentally showing windows while someone else is at the console).