Xtreemconfig < ORUA

Tags: view all tags
---+ !XtreemFS Configuration

There are three core parts of !XtreemFS system:
   * osd - object storage data (actual data storage) 
   * mrc - meta data and replica catalog server (headers storage) 
   * dir - directory (directory service) 

(Note: dir service is similar to 'mon' service in [[CephConfig][Ceph]])

---++ Debian Package Setup

It is recommended to have the Linux kernel 2.6.36 RC 7 (this is the October 2010 kernel).

---+++ Download and Install

We are using *Ubuntu* and the build process is described in detail at external: https://www.xtreemfs.org/quickstart.php

Download the following packages from external: https://www.xtreemfs.org/download.php?t=debs
   * xtreemfs-backend_1.2.3_all.deb 
   * xtreemfs-server_1.2.3_all.deb 
   * xtreemfs-tools_1.2.3_all.deb 
   * xtreemfs-client_1.2.3_amd64.deb 

Install the packages
<verbatim>
$ dpkg -i xtreemfs-client-1.2.3.deb xtreemfs-server-1.2.3.deb xtreemfs-backend-1.2.3.deb
</verbatim> on all the machines (Note. You can only install xtreemfs-server on server machines and xtreemfs-client on client machines, but all machines need xtreemfs-backend and probably xtreemfs-tools).

For details see the single machine install & setup. external https://www.xtreemfs.org/quickstart.php?t=d

---+++ !XtreemFS Server

General note: very important to follow step by step;

Create a directory for OSDs at
<verbatim> /var/lib/xtreemfs/objs/ </verbatim>

*4 Nodes*
   * <strong>s1 - osd0 (s1)<br /></strong> 
   * <strong>s2 - osd1 (s2) <br /></strong> 
   * <strong>s3 - mrc (s3) <br /></strong> 
   * <strong>s4 - dir (s4) <br /></strong> 

plus one additional node for client (in this case, we will use s4). All OSD, MRC and DIR should have the configuration file /etc/xos/xtreemfs/*.properties
   * - dirconfig.properties (dir service) 
   * - mrcconfig.properties (mrc service) 
   * - osdconfig.properties (osd service) 

Since the setup is configured using *.properties, except for the dir service (s4), both osd and mrc must have their respective property file changed to point to the location (IP or hostname) of dir,
   * For MRC (s3), edit /etc/xos/xreemfs/mrcconfig.properties, and change the default line dir_service.host = localhost to dir_service.host = *s4* 
   * For OSD1 and OSD2 (s1 and s2), edit /etc/xos/xreemfs/osdconfig.properties, and change the line dir_service.host = localhost to dir_service.host = *s4* 

Also, you need to make sure you are mounting the OSDs in the correct location
   * For OSD1 and OSD2, make sure to mount the device to "/var/lib/xtreemfs/objs/" as indicated in the line, object_dir = /var/lib/xtreemfs/objs/ It is recommended to mount ext3/ext4 file system. 

Now, start up the servers (dir, mrc, osd). We recommned this order
<verbatim>
   * /etc/init.d/xtreemfs-dir start 
   * /etc/init.d/xtreemfs-mrc start 
   * /etc/init.d/xtreemfs-osd start
</verbatim>

Note that !XtreemFS does not output any errors, so make sure to check the log messages <pre>tail /var/log/xtreemfs/</pre>

Also use <pre>ps -Af | grep xtreemfs</pre> to check that the services are running.

See external https://www.xtreemfs.org/quickstart.php from more details.

---+++ !XtreemFS Client

This part involves two steps:
   1. Create a volume on the MRC-- e.g., Normal, Stripe (RAID0), Replica mode (Note. Volume location is always at *mrc (s3)*.)
   2. Mount the created volume (note - mount location is always at * dir (s4)* 

---++++ Example (Basic FS)
   1. Create FS called 'test' (you must create it to the MRC location)
<verbatim>
$ sudo mkfs.xtreemfs s3/test
</verbatim>

   2. Mount the 'test' (you must mount it to the DIR location, and make sure to create empty directory at /media)
<verbatim>
$ sudo mount.xtreemfs s4/test /media/test
</verbatim>

---++++ Example (RAID-0 for osds on s1 and s2)
<verbatim>
$ sudo mkfs.xtreemfs -p RAID0 -w 2 s3/raid
</verbatim>

   * -p accepts RAID0 only so far (1.2.x) 
   * -w is the 'width' i.e., how many OSDs are to be used for the striping (in round-robin fashion), in this case we have 2 OSDs. 
   * -s is the stripe-size in KB, by default it is 128K (e.g., -s 64 to set to 64KB) 

*Useful additional parameters for mounting*
<verbatim>
$ sudo mount.xtreemfs -o allow_other s4/raid /media/raid 
</verbatim>

<verbatim>
$ sudo mount.xtreemfs --cache-data --metadata-cache -o direct_io -o allow_other s4/raid /media/raid 
</verbatim>

   * --cache-data: enable client-side data caching
   * --cache-metadata: enable client-side metadata caching
   * -o (FUSE option): direct_io, allow_other, big_writes

For more information see
   * external: https://www.xtreemfs.org/xtfs-guide-1.2.1/index.html 
   * man mount.xtreemfs 
   * man mkfs.xtreemfs 

---++++ Check the current OSD configuration
Individual files can have its own file striping methods, e.g.,
<verbatim>
$ sudo xtfs_sp --get /media/test
file:              /media/test
policy:            STRIPING_POLICY_RAID0
stripe-size (kB):  128
width:             1
</verbatim>

---+++ Changing UUID

Each OSD/MRC/DIR must be uniquely identifiable by others.

For OSDs (in our case s1 and s2), each OSDs must have UUID.
<verbatim>
./generate_uuid osdconfig.properties
</verbatim>
Topic revision: r6 - 2012-01-15 - TWikiAdminUser