SystemInstallation < ORUA

Tags: view all tags
---++ System Installation

Example of a single machine's hardware and software installation

   *Hardware* CPU: Intel(R) Xeon CPU E5630 @ 2.53Ghz (8 cores) RAM: DDR3 19GB NIC: Dual GbE Intel Pro PCIe cards (Channel bonding mode 0 (RR) or 4 (LACP))

   *Software* OS: Ubuntu 10.04 LTS x64 / OpenSUSE 11.3 x64

   * username: *aaa* * hostname: *ss1* * IP address: *192.168.1.2* * netmask: *255.255.255.0* * gateway: *192.168.1.254"

   * user names are ' *aaa* ' for all, e.g., aaa@ss1, aaa@ss2, aaa@ss3 and aaa@ss4. * passwords are ' *aaa* ' for all

It is highly recommended to disable sudo passwords for our experiments. In the Terminal, type =sudo visudo=

Find the three lines starting with =root=, =%sudo= and =%admin=, and then replace with the following lines respectively.
<verbatim>
root    ALL=(ALL) NOPASSWD:ALL
%sudo ALL=(ALL) NOPASSWD:ALL
%admin ALL=(ALL) NOPASSWD:ALL
</verbatim>

Now set the network via the =/etc/resolv.conf=, =/etc/hosts=, =/etc/network/interface= files:

*/etc/resolv.conf*
<verbatim>
nameserver 192.168.1.254
</verbatim>

*/etc/hosts*
<verbatim>
127.0.0.1   localhost
127.0.1.1   ss1

192.168.1.1 p5e  #DongJin's machine

# Super-Storage Systems #
192.168.1.2 ss1  #Storage machine 1 - OSD (3U)
192.168.1.3 ss2  #Storage machine 2 - OSD (3U)
192.168.1.4 ss3  #Monitor / MDS machine (2U)
192.168.1.5 ss4  #Client machine (2U)

# Low-Power Systems #
192.168.1.10 lp0  #Monitor / MDS machine (Micro1U SuperMicro)
192.168.1.11 lp1  #Storage machine 1 - OSD (Gigabyte D525TUD)
192.168.1.12 lp2  #Storage machine 2 - OSD (Gigabyte D525TUD)
192.168.1.13 lp3  #Storage machine 3 - OSD (Gigabyte D525TUD)
192.168.1.14 lp4  #Storage machine 4 - OSD (Gigabyte D525TUD)
192.168.1.15 lp5  #Storage machine 5 - OSD (Gigabyte D525TUD)
192.168.1.16 lp6  #Storage machine 6 - OSD (Gigabyte D525TUD)

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
</verbatim>

*/etc/network/interfaces*

<verbatim>
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.254
mtu 9000
bond-slaves eth0 eth1
bond-mode 0
bond-miimon 100   

auto eth0
iface eth0 inet static
bond-master bond0
bond-primary eth0 eth1

auto eth1
iface eth1 inet static
bond-master bond0
bond-primary eth0 eth1
</verbatim> and restart the networking:

=sudo /etc/init.d/networking restart=

---+++ Preliminarily packages

Following packages should be installed:
   * git (for retrieving and updating Ceph repository) 
   * ifenslave 2.6 (channel bonding) 
   * bmon (monitoring the individual NICs for checking / estimating the rate) 
   * sysinfo (system information) 
   * mdadm (RAID) 
   * sysstat (iostat) 
   * iperf (traffic sending/receiving, useful for finding bottleneck/bandwidth) 
   * ssh (ssh) 
   * wireshark (capturing packets,etc) 
   * traceroute (checking packet routes) 
   * dia (for drawing system diagrams) 
   * libgtkmm (for Ceph GUI) 


Finally check that the system is running as expected - check if the bonding is working correctly, use =iperf= to test throughputs are expected (i.e., 1Gb/s, 2Gb/s) are observed. - Also check bonding as shown below.

<verbatim>
aaa@ss1:~$ cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.0 (June 2, 2010)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:25:90:18:14:80
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Link Failure Count: 3
Permanent HW addr: 00:25:90:18:14:81
Slave queue ID: 0
</verbatim>

---+++ Setting up ssh (no) password

Since the system will use ssh and scp operations in many scripts, it is recommended to bypass the ssh password: this is done by using public/private keying.

As an example, if *ss4* (Ceph Client) wants to access for ssh/scp to *ss1* (Ceph OSD), then *ss4* first needs to have its own 'public-key' stored in *ss1*.

The quickest way is to generate private/public key from ss4, copy the public-key to ss1, as below.

<verbatim>
aaa@ss4:~$ ssh-keygen -t rsa
</verbatim> There's no need to enter pass-phrase when asked (just type enter twice). The command generates two files (private + public keys) that are stored in the =/home/aaa/.ssh/= directory. Now simply type as below to copy the ss3's public key to ss1 <verbatim>
aaa@ss4:~$ ssh-copy-id ss1
</verbatim>

Type 'yes' when it ask to store the address into 'known-hosts' list. Vice versa if *ss1* would like to access *ss4*. (At this stage, our client)

Also it is important to make sure that the username is explicitly stated if both systems use different username. In the above example, there was no need to state *aaa@ss1* because both systems use the same username. If you want to access system such as *mike@bb1*, from *ss4*, then you need to make sure to enter full name; *mike@bb1* every time. Without typing the username, it will try to to access as *aaa@bb1* which *bb1* does not have (yet, unless you separately create one).

---+++ OpenSUSE 11.3 System configuration
We also use OpenSUSE as our OS. It uses RPM packages instead of DEB.
The configurations are similar but some differences are found. See below.

   * No longer user name 'aaa' -- for simplicity, all are 'root' with super-user privilege, making it easier for the ceph configurations (but be careful!)
   * Configuring the NIC and Bond0 (follow the GUI) --> much more reliable bonding than Debian (it is less error-prone)
Instead of a single file with all of the NIC */etc/network/interfaces* (Debian/Ubuntu), openSUSE uses each NIC name in a separate file.

For example, configuring eth0, *etc/sysconfig/network/ifcfg-eth0*
<verbatim>
BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS='-K eth2 rx off tx off sc off tso off udo off gso on'
IPADDR='192.168.1.16/24'
MTU=''
NAME='RTL8111/8168B PCI Express Gigabit Ethernet controller'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
</verbatim>

Also, the gateway address is in a separate file, */etc/sysconfig/network/routes*
<verbatim>
default 192.168.1.254 - eth2
</verbatim>

And lastly,  */etc/HOSTNAME* is a capital (the hostname), 
 
   
---+++ Tuning Networks

For OpenSuSe 
<verbatim> 
sysctl -a | egrep 'rmem|wmem|tcp_congestion|reordering|metrics' | egrep -v 'default|udp|lowmem'
</verbatim>

copy the following linesto "/etc/sysctl.conf"

<verbatim>
net.ipv4.tcp_reordering = 20
net.ipv4.tcp_wmem = 8192 87380 16777216
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_congestion_control = cubic
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
</verbatim>

and finally (re)start the sysctl,

<verbatim>
sysctl -p /etc/sysctl.conf
</verbatim>
Refer to  https://www.susegeek.com/networking/network-performance-fine-tuning-in-opensuse-suse/
Topic revision: r1 - 2011-09-15 - DongJinLee