System Installation

Example of a single machine's hardware and software installation

Hardware CPU: Intel(R) Xeon CPU E5630 @ 2.53Ghz (8 cores) RAM: DDR3 19GB NIC: Dual GbE Intel Pro PCIe cards (Channel bonding mode 0 (RR) or 4 (LACP))

Software OS: Ubuntu 10.04 LTS x64 / OpenSUSE 11.3 x64

• username: aaa * hostname: ss1 * IP address: 192.168.1.2 * netmask: 255.255.255.0 * gateway: *192.168.1.254"

• user names are ' aaa ' for all, e.g., aaa@ss1, aaa@ss2, aaa@ss3 and aaa@ss4. * passwords are ' aaa ' for all

It is highly recommended to disable sudo passwords for our experiments. In the Terminal, type sudo visudo

Find the three lines starting with root, %sudo and %admin, and then replace with the following lines respectively.

root    ALL=(ALL) NOPASSWD:ALL
%sudo ALL=(ALL) NOPASSWD:ALL
%admin ALL=(ALL) NOPASSWD:ALL

Now set the network via the /etc/resolv.conf, /etc/hosts, /etc/network/interface files:

/etc/resolv.conf

nameserver 192.168.1.254

/etc/hosts

127.0.0.1   localhost
127.0.1.1   ss1

192.168.1.1 p5e  #DongJin's machine

# Super-Storage Systems #
192.168.1.2 ss1  #Storage machine 1 - OSD (3U)
192.168.1.3 ss2  #Storage machine 2 - OSD (3U)
192.168.1.4 ss3  #Monitor / MDS machine (2U)
192.168.1.5 ss4  #Client machine (2U)

# Low-Power Systems #
192.168.1.10 lp0  #Monitor / MDS machine (Micro1U SuperMicro)
192.168.1.11 lp1  #Storage machine 1 - OSD (Gigabyte D525TUD)
192.168.1.12 lp2  #Storage machine 2 - OSD (Gigabyte D525TUD)
192.168.1.13 lp3  #Storage machine 3 - OSD (Gigabyte D525TUD)
192.168.1.14 lp4  #Storage machine 4 - OSD (Gigabyte D525TUD)
192.168.1.15 lp5  #Storage machine 5 - OSD (Gigabyte D525TUD)
192.168.1.16 lp6  #Storage machine 6 - OSD (Gigabyte D525TUD)

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

/etc/network/interfaces

auto lo
iface lo inet loopback

auto bond0
iface bond0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.254
mtu 9000
bond-slaves eth0 eth1
bond-mode 0
bond-miimon 100   

auto eth0
iface eth0 inet static
bond-master bond0
bond-primary eth0 eth1

auto eth1
iface eth1 inet static
bond-master bond0
bond-primary eth0 eth1

and restart the networking:

sudo /etc/init.d/networking restart

Preliminarily packages

Following packages should be installed:

• git (for retrieving and updating Ceph repository)
• ifenslave 2.6 (channel bonding)
• bmon (monitoring the individual NICs for checking / estimating the rate)
• sysinfo (system information)
• mdadm (RAID)
• sysstat (iostat)
• iperf (traffic sending/receiving, useful for finding bottleneck/bandwidth)
• ssh (ssh)
• wireshark (capturing packets,etc)
• traceroute (checking packet routes)
• dia (for drawing system diagrams)
• libgtkmm (for Ceph GUI)

Finally check that the system is running as expected - check if the bonding is working correctly, use iperf to test throughputs are expected (i.e., 1Gb/s, 2Gb/s) are observed. - Also check bonding as shown below.

aaa@ss1:~$ cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.0 (June 2, 2010)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:25:90:18:14:80
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Link Failure Count: 3
Permanent HW addr: 00:25:90:18:14:81
Slave queue ID: 0

Setting up ssh (no) password

Since the system will use ssh and scp operations in many scripts, it is recommended to bypass the ssh password: this is done by using public/private keying.

As an example, if ss4 (Ceph Client) wants to access for ssh/scp to ss1 (Ceph OSD), then ss4 first needs to have its own 'public-key' stored in ss1.

The quickest way is to generate private/public key from ss4, copy the public-key to ss1, as below.

aaa@ss4:~$ ssh-keygen -t rsa

There's no need to enter pass-phrase when asked (just type enter twice). The command generates two files (private + public keys) that are stored in the /home/aaa/.ssh/ directory. Now simply type as below to copy the ss3's public key to ss1

aaa@ss4:~$ ssh-copy-id ss1

Type 'yes' when it ask to store the address into 'known-hosts' list. Vice versa if ss1 would like to access ss4. (At this stage, our client)

Also it is important to make sure that the username is explicitly stated if both systems use different username. In the above example, there was no need to state aaa@ss1 because both systems use the same username. If you want to access system such as mike@bb1, from ss4, then you need to make sure to enter full name; mike@bb1 every time. Without typing the username, it will try to to access as aaa@bb1 which bb1 does not have (yet, unless you separately create one).

OpenSUSE 11.3 System configuration

We also use OpenSUSE as our OS. It uses RPM packages instead of DEB. The configurations are similar but some differences are found. See below.

• No longer user name 'aaa' -- for simplicity, all are 'root' with super-user privilege, making it easier for the ceph configurations (but be careful!)
• Configuring the NIC and Bond0 (follow the GUI) --> much more reliable bonding than Debian (it is less error-prone)

Instead of a single file with all of the NIC /etc/network/interfaces (Debian/Ubuntu), openSUSE uses each NIC name in a separate file.

For example, configuring eth0, etc/sysconfig/network/ifcfg-eth0

BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS='-K eth2 rx off tx off sc off tso off udo off gso on'
IPADDR='192.168.1.16/24'
MTU=''
NAME='RTL8111/8168B PCI Express Gigabit Ethernet controller'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'

Also, the gateway address is in a separate file, /etc/sysconfig/network/routes

default 192.168.1.254 - eth2

And lastly, /etc/HOSTNAME is a capital (the hostname),

Tuning Networks

For OpenSuSe

 
sysctl -a | egrep 'rmem|wmem|tcp_congestion|reordering|metrics' | egrep -v 'default|udp|lowmem'

copy the following linesto "/etc/sysctl.conf"

net.ipv4.tcp_reordering = 20
net.ipv4.tcp_wmem = 8192 87380 16777216
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_congestion_control = cubic
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216

and finally (re)start the sysctl,

sysctl -p /etc/sysctl.conf

Refer to https://www.susegeek.com/networking/network-performance-fine-tuning-in-opensuse-suse/