Difference: Xtreemconfig (1 vs. 6)

Revision 62012-01-15 - TWikiAdminUser

Line: 1 to 1
 
META TOPICPARENT name="ORUA.ExperimentalStorage"

XtreemFS Configuration

Line: 7 to 7
 
  • mrc - meta data and replica catalog server (headers storage)
  • dir - directory (directory service)
Changed:
<
<
(Note: dir service is similar to 'mon' service in Ceph)
>
>
(Note: dir service is similar to 'mon' service in Ceph)
 

Debian Package Setup

Line: 130 to 130
 
./generate_uuid osdconfig.properties
\ No newline at end of file
Added:
>
>
META TOPICMOVED by="BaseUserMapping_333" date="1326592876" from="NDSG.Xtreemconfig" to="ORUA.Xtreemconfig"

Revision 52011-09-08 - TWikiAdminUser

Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="StorageImplementation"
>
>
META TOPICPARENT name="ORUA.ExperimentalStorage"
 

XtreemFS Configuration

There are three core parts of XtreemFS system:

Revision 42011-01-12 - MichaelOSullivan

Line: 1 to 1
 
META TOPICPARENT name="StorageImplementation"
Deleted:
<
<
 

XtreemFS Configuration

There are three core parts of XtreemFS system:

Revision 32011-01-07 - DongJinLee

Line: 1 to 1
 
META TOPICPARENT name="StorageImplementation"
Added:
>
>
 

XtreemFS Configuration

There are three core parts of XtreemFS system:

Revision 22010-10-18 - DongJinLee

Line: 1 to 1
 
META TOPICPARENT name="StorageImplementation"

XtreemFS Configuration

Line: 26 to 26
 Install the packages
$ dpkg -i xtreemfs-client-1.2.3.deb xtreemfs-server-1.2.3.deb xtreemfs-backend-1.2.3.deb
Changed:
<
<
on all the machines (Note. You can only install xtreemfs-server on server machines and xtreemfs-client on client machines, but all machines need xtreemfs-backend and probably xtreemfs-tools).
>
>
on all the machines (Note. You can only install xtreemfs-server on server machines and xtreemfs-client on client machines, but all machines need xtreemfs-backend and probably xtreemfs-tools).
  For details see the single machine install & setup. external https://www.xtreemfs.org/quickstart.php?t=d
Line: 35 to 34
  General note: very important to follow step by step;
Changed:
<
<
Create a directory for OSDs
 /var/lib/xtreemfs/objs/
.
>
>
Create a directory for OSDs at
 /var/lib/xtreemfs/objs/ 
  4 Nodes
  • s1 - osd0 (s1)
Line: 43 to 43
 
  • s3 - mrc (s3)
  • s4 - dir (s4)
Changed:
<
<
plus one additional node for client (in this case, we will use s4). All OSD, MRC and DIR should have the configuration file /etc/xos/xtreemfs/*.properties - dirconfig.properties (dir service) - mrcconfig.properties (mrc service) - osdproperties (osd service).

Since the setup is configued using *.properties, except for the dir service (s4), both osd and mrc must have their respective property file changed to point to the location (IP or hostname) of dir,

  • For MRC (s3), edit /etc/xos/xreemfs/mrcconfig.properties, and change the default line dir_service.host = localhost to dir_service.host = s4
  • For OSD1 and OSD2 (s1 and s2), edit /etc/xos/xreemfs/osdconfig.properties, and change the line dir_service.host = localhost to dir_service.host = s4 *
>
>
plus one additional node for client (in this case, we will use s4). All OSD, MRC and DIR should have the configuration file /etc/xos/xtreemfs/*.properties
  • - dirconfig.properties (dir service)
  • - mrcconfig.properties (mrc service)
  • - osdconfig.properties (osd service)

Since the setup is configured using *.properties, except for the dir service (s4), both osd and mrc must have their respective property file changed to point to the location (IP or hostname) of dir,

  • For MRC (s3), edit /etc/xos/xreemfs/mrcconfig.properties, and change the default line dir_service.host = localhost to dir_service.host = s4
  • For OSD1 and OSD2 (s1 and s2), edit /etc/xos/xreemfs/osdconfig.properties, and change the line dir_service.host = localhost to dir_service.host = s4
  Also, you need to make sure you are mounting the OSDs in the correct location
  • For OSD1 and OSD2, make sure to mount the device to "/var/lib/xtreemfs/objs/" as indicated in the line, object_dir = /var/lib/xtreemfs/objs/ It is recommended to mount ext3/ext4 file system.

Now, start up the servers (dir, mrc, osd). We recommned this order

Changed:
<
<
>
>
 
  • /etc/init.d/xtreemfs-dir start
  • /etc/init.d/xtreemfs-mrc start
  • /etc/init.d/xtreemfs-osd start
Added:
>
>

Note that XtreemFS does not output any errors, so make sure to check the log messages

tail /var/log/xtreemfs/
 
Changed:
<
<
make sure to watch out for any errors.
>
>
Also use
ps -Af | grep xtreemfs
to check that the services are running.
  See external https://www.xtreemfs.org/quickstart.php from more details.

XtreemFS Client

Changed:
<
<
This is the where you mount and use XtreemFS:
>
>
This part involves two steps:
  1. Create a volume on the MRC-- e.g., Normal, Stripe (RAID0), Replica mode (Note. Volume location is always at mrc (s3).)
  2. Mount the created volume (note - mount location is always at * dir (s4)*

Example (Basic FS)

  1. Create FS called 'test' (you must create it to the MRC location)
$ sudo mkfs.xtreemfs s3/test

  1. Mount the 'test' (you must mount it to the DIR location, and make sure to create empty directory at /media)
$ sudo mount.xtreemfs s4/test /media/test

Example (RAID-0 for osds on s1 and s2)

$ sudo mkfs.xtreemfs -p RAID0 -w 2 s3/raid

  • -p accepts RAID0 only so far (1.2.x)
  • -w is the 'width' i.e., how many OSDs are to be used for the striping (in round-robin fashion), in this case we have 2 OSDs.
  • -s is the stripe-size in KB, by default it is 128K (e.g., -s 64 to set to 64KB)

Useful additional parameters for mounting

$ sudo mount.xtreemfs -o allow_other s4/raid /media/raid 

$ sudo mount.xtreemfs --cache-data --metadata-cache -o direct_io -o allow_other s4/raid /media/raid 

  • --cache-data: enable client-side data caching
  • --cache-metadata: enable client-side metadata caching
  • -o (FUSE option): direct_io, allow_other, big_writes

For more information see

Check the current OSD configuration

Individual files can have its own file striping methods, e.g.,
$ sudo xtfs_sp --get /media/test
file:              /media/test
policy:            STRIPING_POLICY_RAID0
stripe-size (kB):  128
width:             1

Changing UUID

Each OSD/MRC/DIR must be uniquely identifiable by others.

For OSDs (in our case s1 and s2), each OSDs must have UUID.

./generate_uuid osdconfig.properties

Revision 12010-10-17 - DongJinLee

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="StorageImplementation"

XtreemFS Configuration

There are three core parts of XtreemFS system:

  • osd - object storage data (actual data storage)
  • mrc - meta data and replica catalog server (headers storage)
  • dir - directory (directory service)

(Note: dir service is similar to 'mon' service in Ceph)

Debian Package Setup

It is recommended to have the Linux kernel 2.6.36 RC 7 (this is the October 2010 kernel).

Download and Install

We are using Ubuntu and the build process is described in detail at external: https://www.xtreemfs.org/quickstart.php

Download the following packages from external: https://www.xtreemfs.org/download.php?t=debs * xtreemfs-backend_1.2.3_all.deb * xtreemfs-server_1.2.3_all.deb * xtreemfs-tools_1.2.3_all.deb * xtreemfs-client_1.2.3_amd64.deb

Install the packages

$ dpkg -i xtreemfs-client-1.2.3.deb xtreemfs-server-1.2.3.deb xtreemfs-backend-1.2.3.deb
on all the machines (Note. You can only install xtreemfs-server on server machines and xtreemfs-client on client machines, but all machines need xtreemfs-backend and probably xtreemfs-tools).

For details see the single machine install & setup. external https://www.xtreemfs.org/quickstart.php?t=d

XtreemFS Server

General note: very important to follow step by step;

Create a directory for OSDs

 /var/lib/xtreemfs/objs/
.

4 Nodes

  • s1 - osd0 (s1)
  • s2 - osd1 (s2)
  • s3 - mrc (s3)
  • s4 - dir (s4)

plus one additional node for client (in this case, we will use s4). All OSD, MRC and DIR should have the configuration file /etc/xos/xtreemfs/*.properties - dirconfig.properties (dir service) - mrcconfig.properties (mrc service) - osdproperties (osd service).

Since the setup is configued using *.properties, except for the dir service (s4), both osd and mrc must have their respective property file changed to point to the location (IP or hostname) of dir,

  • For MRC (s3), edit /etc/xos/xreemfs/mrcconfig.properties, and change the default line dir_service.host = localhost to dir_service.host = s4
  • For OSD1 and OSD2 (s1 and s2), edit /etc/xos/xreemfs/osdconfig.properties, and change the line dir_service.host = localhost to dir_service.host = s4 *

Also, you need to make sure you are mounting the OSDs in the correct location

  • For OSD1 and OSD2, make sure to mount the device to "/var/lib/xtreemfs/objs/" as indicated in the line, object_dir = /var/lib/xtreemfs/objs/ It is recommended to mount ext3/ext4 file system.

Now, start up the servers (dir, mrc, osd). We recommned this order

  • /etc/init.d/xtreemfs-dir start
  • /etc/init.d/xtreemfs-mrc start
  • /etc/init.d/xtreemfs-osd start

make sure to watch out for any errors.

See external https://www.xtreemfs.org/quickstart.php from more details.

XtreemFS Client

This is the where you mount and use XtreemFS:

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback