字里行间

Train Overview

Fri, 25 Oct 2019 00:00:00 +0000

This is the Example of building Cloud Computing infrastracture by OpenStack Train. Please learn simply description about OpenStack below before building.

(1) Main Components of OpenStack

Service

Code Name

Description

Identity Service

Keystone

User Management

Compute Service

Nova

Virtual Machine Management

Image Service

Glance

Manages Virtual image like kernel image or disk image

Dashboard

Horizon

Provides GUI console via Web browser

Object Storage

Swift

Provides Cloud Storage

Block Storage

Cinder

Download CentOS 7

Tue, 17 Sep 2019 00:00:00 +0000

This is the configuration Examples for CentOS 7.x that is compatible with Red Hat Enterprise Linux 7.x. CentOS 7 was released on 7 July 2014 and will be supported untill the end of June, 2024.

Download the CentOS 7.9 ISO file for installation from the following site.

⇒ http://isoredirect.centos.org/centos/7.9.2009/isos/x86_64/

Create an Installation disk for CentOS 7

After downloading ISO of CentOS 7.x, Burn it to a DVD with an burning application to create an installation disk. If you create it with the function of Windows 7/8/10, refer to follows.

Install Dnsmasq

Wed, 31 Aug 2016 00:00:00 +0000

Install Dnsmasq which is the lightweight DNS forwarder and DHCP Server Software.

Install Dnsmasq.

[root@dlp ~]# yum -y install dnsmasq

Configure Dnsmasq.

[root@dlp ~]# vi /etc/dnsmasq.conf
# line 19: uncomment (never forward plain names)
domain-needed
# line 21: uncomment (never forward addresses in the non-routed address spaces)
bogus-priv
# line 41: uncomment (query with each server strictly in the order in resolv.conf)
strict-order
# line 55: add if you need
# query the specific domain name to the specific DNS server
# the example follows means query [server.education] domain to the [10.0.0.10] server
server=/server.education/10.0.0.10
# line 123: uncomment (add domain name automatically)
expand-hosts
# line 133: add (define domain name)
domain=srv.world
[root@dlp ~]# systemctl start dnsmasq
[root@dlp ~]# systemctl enable dnsmasq

For DNS records, add them in /etc/hosts.

Configure NFS Server

Sat, 10 Jan 2015 00:00:00 +0000

Configure NFS Server to share directories on your Network. This example is based on the environment below.

+----------------------+ | +----------------------+
| [ NFS Server ] |10.0.0.30 | 10.0.0.31| [ NFS Client ] |
| dlp.srv.world +----------+----------+ www.srv.world |
| | | |
+----------------------+ +----------------------+

Configure NFS Server.

[root@dlp ~]# yum -y install nfs-utils
[root@dlp ~]# vi /etc/idmapd.conf
# line 5: uncomment and change to your domain name
Domain =
srv.world
[root@dlp ~]# vi /etc/exports
# write settings for NFS exports
/home 10.0.0.0/24(rw,no_root_squash)
[root@dlp ~]# systemctl start rpcbind nfs-server
[root@dlp ~]# systemctl enable rpcbind nfs-server

If Firewalld is running

Install KVM

Wed, 07 Jan 2015 00:00:00 +0000

It’s Virtualization with KVM ( Kernel-based Virtual Machine ) + QEMU. This requires that the CPU on your computer which has a function Intel VT or AMD-V.

Install KVM.

[root@dlp ~]# yum -y install qemu-kvm libvirt virt-install bridge-utils
# make sure modules are loaded
[root@dlp ~]# lsmod | grep kvm
kvm_intel 138567 0
kvm 441119 1 kvm_intel
[root@dlp ~]# systemctl start libvirtd
[root@dlp ~]# systemctl enable libvirtd

Configure Bridge networking for KVM virtual machine.

Configure NTP Server (NTPd)

Tue, 08 Jul 2014 00:00:00 +0000

Install NTPd and Configure NTP server for time adjustment.

Install NTPd.

[root@dlp ~]# yum -y install ntp
[root@dlp ~]# vi /etc/ntp.conf
# line 18: add the network range you allow to receive requests
restrict 10.0.0.0 mask 255.255.255.0 nomodify notrap
# change servers for synchronization
#
server 0.centos.pool.ntp.org iburst
#
server 1.centos.pool.ntp.org iburst
#
server 2.centos.pool.ntp.org iburst
#
server 3.centos.pool.ntp.org iburst
server ntp1.jst.mfeed.ad.jp iburst server ntp2.jst.mfeed.ad.jp iburst server ntp3.jst.mfeed.ad.jp iburst
[root@dlp ~]# systemctl start ntpd
[root@dlp ~]# systemctl enable ntpd

If Firewalld is running

Pre-Requirements

Fri, 25 Oct 2019 00:00:00 +0000

This is the example of Cloud Computiong by OpenStack Train.

Install some services that some components of OpenStack needs for system requirements on here. This example is based on the emvironment like follows.

eth0|10.0.0.30 
+-----------+-----------+
| [ Control Node ] |
| |
| MariaDB RabbitMQ |
| Memcached |
+-----------------------+

Install NTP Server to adjusts the date, refer to here .

Add Repository of Openstack Train.

[root@dlp ~]# yum -y install centos-release-openstack-train
[root@dlp ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-OpenStack-train.repo

Install MariaDB server.

Configure DHCP Server

Wed, 31 Aug 2016 00:00:00 +0000

Enable integrated DHCP function in Dnsmasq and Configure DHCP Server.

Configure Dnsmasq.

[root@dlp ~]# vi /etc/dnsmasq.conf
# line 146: add (range of IP address to lease and term of lease)
dhcp-range=10.0.0.200,10.0.0.250,12h
# line 316: add (define default gateway)
dhcp-option=option:router,10.0.0.1
# line 325: add (define NTP, DNS, server and subnetmask)
dhcp-option=option:ntp-server,10.0.0.10 dhcp-option=option:dns-server,10.0.0.10 dhcp-option=option:netmask,255.255.255.0
[root@dlp ~]# systemctl restart dnsmasq

If Firewalld is running

, allow DHCP service. DHCP server uses 67/UDP.

[root@dlp ~]# firewall-cmd --add-service=dhcp --permanent
success
[root@dlp ~]# firewall-cmd --reload
success

It’s OK all, Configure DHCP client on client computer and verify it works normally.

Configure NTP Server (Chrony)

Wed, 03 Jun 2015 00:00:00 +0000

Install Chrony and Configure NTP server for time adjustment.

Install Chrony.

[root@dlp ~]# yum -y install chrony
[root@dlp ~]# vi /etc/chrony.conf
# line 3: change servers for synchronization
#
server 0.centos.pool.ntp.org iburst
#
server 1.centos.pool.ntp.org iburst
#
server 2.centos.pool.ntp.org iburst
#
server 3.centos.pool.ntp.org iburst
server ntp1.jst.mfeed.ad.jp iburst server ntp2.jst.mfeed.ad.jp iburst server ntp3.jst.mfeed.ad.jp iburst
# line 25: add the network range you allow to receive requests
allow 10.0.0.0/24
[root@dlp ~]# systemctl start chronyd
[root@dlp ~]# systemctl enable chronyd

If Firewalld is running

Conf NFS Client(CentOS)

Sat, 10 Jan 2015 00:00:00 +0000

This example is based on the environment below.

+----------------------+ | +----------------------+
| [ NFS Server ] |10.0.0.30 | 10.0.0.31| [ NFS Client ] |
| dlp.srv.world +----------+----------+ www.srv.world |
| | | |
+----------------------+ +----------------------+

Configure NFS Client.

[root@www ~]# yum -y install nfs-utils
[root@www ~]# vi /etc/idmapd.conf
# line 5: uncomment and change to your domain name
Domain =
srv.world
[root@www ~]# systemctl start rpcbind
[root@www ~]# systemctl enable rpcbind
[root@www ~]# mount -t nfs dlp.srv.world:/home /home
[root@www ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/centos-root xfs 46G 1.4G 45G 4% /
devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs tmpfs 1.9G 8.3M 1.9G 1% /run
tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vda1 xfs 497M 219M 278M 45% /boot
dlp.srv.world:/home nfs4 46G 1.4G 45G 4% /home
# /home from NFS server is mounted
# if you like to mount with NFSv3, add the '-o vers=3' option
[root@www ~]# mount -t nfs -o vers=3 dlp.srv.world:/home /home

Configure NFS mounting on fstab to mount it when the system boot.

Create a Virtual Machine

Fri, 11 Jul 2014 00:00:00 +0000

Install GuestOS and create a Virtual Machine. This example shows to install CentOS 7.

Install GuestOS on text mode via network, it’s OK on Console or remote connection with Putty and so on. Furthermore, Virtual Machine’s images are placed at /var/lib/libvirt/images by default as a Storage Pool, but this example shows to create and use a new Storage Pool.

[root@dlp ~]# mkdir -p /var/kvm/images
# create a new Storage Pool
[root@dlp ~]# virt-install \ --name centos7 \ --ram 4096 \ --disk path=/var/kvm/images/centos7.img,size=30 \ --vcpus 2 \ --os-type linux \ --os-variant rhel7 \ --network bridge=br0 \ --graphics none \ --console pty,target_type=serial \ --location 'http://ftp.iij.ad.jp/pub/linux/centos/7/os/x86_64/' \ --extra-args 'console=ttyS0,115200n8 serial'
Starting install...
# start installation

The example of options above means like follows. There are many options for others, make sure with “man virt-install”.

Install CentOS 7

Tue, 08 Jul 2014 00:00:00 +0000

Insert CentOS install disk and start the Computer. Then, Following screen is displayed, Push Enter key to proceed.

Select the language you’d like to use during the installation.

This is the default screen for some basic configuration. First, set timezone, click the [DATE&TIME] icon.

Click a point on the map you’d like to set your timezone and push [Done] button which is on the upper left.

Back to the default screen. Next, click the [KEYBOARD] icon.

Configure for Internal Network

Wed, 24 Feb 2021 00:00:00 +0000

Install BIND to Configure DNS (Domain Name System) Server to provide Name or Address Resolution service for Clients.

Install BIND.

[root@dlp ~]# yum -y install bind bind-utils

On this example, Configure BIND for Internal Network.

The example follows is for the case that Local network is [10.0.0.0/24], Domain name is [srv.world], Replace them to your own environment.

[root@dlp ~]# vi /etc/named.conf
.....
.....
# add : set ACL entry for local network
acl internal-network {
 10.0.0.0/24;
};

options {
 # change ( listen all )
 listen-on port 53 { any; };
 # change if need ( if not listen IPv6, set [none] )
 listen-on-v6 { any; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 secroots-file "/var/named/data/named.secroots";
 recursing-file "/var/named/data/named.recursing";
 # add local network set on [acl] section above
 # network range you allow to receive queries from hosts
 allow-query { localhost; internal-network; };
 # network range you allow to transfer zone files to clients
 # add secondary DNS servers if it exist
 allow-transfer { localhost; };

 .....
 .....

 recursion yes;

 dnssec-enable yes;
 dnssec-validation yes;

 managed-keys-directory "/var/named/dynamic";

 pid-file "/run/named/named.pid";
 session-keyfile "/run/named/session.key";

 /* https://fedoraproject.org/wiki/Changes/CryptoPolicy */
 include "/etc/crypto-policies/back-ends/bind.config";
};

logging {
 channel default_debug {
 file "data/named.run";
 severity dynamic;
 };
};

zone "." IN {
 type hint;
 file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

# add zones for your network and domain name
zone "srv.world" IN {
 type master;
 file "srv.world.lan";
 allow-update { none; };
};
zone "0.0.10.in-addr.arpa" IN {
 type master;
 file "0.0.10.db";
 allow-update { none; };
};
# if you don't use IPv6 and also suppress logs for IPv6 related, possible to change
# set BIND to use only IPv4
[root@dlp ~]# vi /etc/sysconfig/named
# add to the end OPTIONS="-4"
# For how to write the section [*.*.*.*.in-addr.arpa], write your network address reversely like follows
# case of 10.0.0.0/24
# network address ⇒ 10.0.0.0
# network range ⇒ 10.0.0.0 - 10.0.0.255
# how to write ⇒ 0.0.10.in-addr.arpa
# 
# case of 192.168.1.0/24
# network address ⇒ 192.168.1.0
# network range ⇒ 192.168.1.0 - 192.168.1.255
# how to write ⇒ 1.168.192.in-addr.arpa

Next, Configure Zone Files for each Zone you set in [named.conf] above. To Configure Zone Files, refer to here .

Configure Keystone #1

Fri, 25 Oct 2019 00:00:00 +0000

Install and Configure OpenStack Identity Service (Keystone).

This example is based on the emvironment like follows.

eth0|10.0.0.30 
+-----------+-----------+
| [ Control Node ] |
| |
| MariaDB RabbitMQ |
| Memcached httpd |
| Keystone |
+-----------------------+

Add a User and Database on MariaDB for Keystone.

[root@dlp ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.3.10-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
create database keystone;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]>
grant all privileges on keystone.* to keystone@'localhost' identified by 'password';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]>
grant all privileges on keystone.* to keystone@'%' identified by 'password';
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]>
flush privileges;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]>
exit
Bye

Install Keystone.

NFS 4 ACL Tool

Fri, 21 Dec 2018 00:00:00 +0000

It’s possible to set ACL on NFS(v4) filesystem to install NFS 4 ACL tool. For setting ACL of XFS or ext4 filesystem, refer to here .

Install NFS 4 ACL Tool on NFS clients that mounts NFS share with NFSv4.

[root@node01 ~]# yum -y install nfs4-acl-tools

On this example, Configure on the environment like follows.

[root@node01 ~]# df -hT /mnt
Filesystem Type Size Used Avail Use% Mounted on
dlp.srv.world:/var/lib/nfs/share nfs4 71G 1.3G 70G 2% /mnt
[root@node01 ~]# ll /mnt
total 4
drwx------. 2 root root 26 Dec 20 19:12 testdir
-rwx------. 1 root root 10 Dec 20 19:11 test.txt

Show ACL of a file or directory on NFSv4 filesystem.

Create a Virtual Machine (GUI)

Fri, 11 Jul 2014 00:00:00 +0000

If you installed Desktop Environment , it’s possible to create a Virtual machine on GUI. This example shows to install Windows Server 2012 R2 on GUI.

Install virt-manager first.

[root@dlp ~]# yum -y install virt-manager

Start Desktop and run ‘Virtual Machine Manager’.

Click ‘New’ button (it’s just the PC-icon on upper-left) and open a wizard to create a new Virtual Machine.

Specify the name of virtual machine and installation source. This example selects local media.

Add an User

Tue, 08 Jul 2014 00:00:00 +0000

Add an User “cent” as an example below.

[root@dlp ~]# useradd cent
[root@dlp ~]# passwd cent
Changing password for user cent.
New UNIX password:
# set password
Retype new UNIX password:
# confirm
passwd: all authentication tokens updated successfully.
[root@dlp ~]# exit
# logout

Try to switch to a user that was added above.

dlp login:
cent
# input user name
password:
# password
[cent@dlp ~]$ su -
# switch to root
Password:
# root password
# just switched to root

Make a user be only a user who can switch to root as an administration user. (it’s ‘cent’ in this example)

Configure NTP Client

Tue, 08 Jul 2014 00:00:00 +0000

Configure NTP Client.

The settings of NTP Client on CentOS are mostly the same with Server’s settings, so refer to NTPd Settings or Chrony Settings . For different settings from Server’s one, Clients don’t need to receive time synchronization requests from other computers, so it does not need to set access permission.

If you don’t use NTP service daemon but use a command to sync time at once, use ntpdate like follows.

Configure for External Network

Wed, 24 Feb 2021 00:00:00 +0000

Install BIND to Configure DNS (Domain Name System) Server to provide Name or Address Resolution service for Clients.

Install BIND.

[root@dlp ~]# yum -y install bind bind-utils

On this example, Configure BIND for External Network.

The example follows is for the case that External network is [172.16.0.80/29], Domain name is [srv.world], Replace them to your own environment.

( Actually, [172.16.0.80/29] is for private IP addresses, though. )

[root@dlp ~]# vi /etc/named.conf
.....
.....
options {
 # change ( listen all )
 listen-on port 53 { any; };
 # change if need ( if not listen IPv6, set [none] )
 listen-on-v6 { any; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 secroots-file "/var/named/data/named.secroots";
 recursing-file "/var/named/data/named.recursing";
 # change : receive queries from all hosts
 allow-query { any; };
 # network range you allow to transfer zone files to clients
 # add secondary DNS servers if it exist
 allow-transfer { localhost; };

 .....
 .....

 # change : not allow recursive queries
 # answer to zones only this server has their entries
 recursion no;

 dnssec-enable yes;
 dnssec-validation yes;

 managed-keys-directory "/var/named/dynamic";

 pid-file "/run/named/named.pid";
 session-keyfile "/run/named/session.key";

 /* https://fedoraproject.org/wiki/Changes/CryptoPolicy */
 include "/etc/crypto-policies/back-ends/bind.config";
};

logging {
 channel default_debug {
 file "data/named.run";
 severity dynamic;
 };
};

zone "." IN {
 type hint;
 file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

# add zones for your network and domain name
zone "srv.world" IN {
 type master;
 file "srv.world.wan";
 allow-update { none; };
};
zone "0.16.172.in-addr.arpa" IN {
 type master;
 file "0.16.172.db";
 allow-update { none; };
};
# if you don't use IPv6 and also suppress logs for IPv6 related, possible to change
# set BIND to use only IPv4
[root@dlp ~]# vi /etc/sysconfig/named
# add to the end OPTIONS="-4"

Next, Configure Zone Files for each Zone you set in [named.conf] above. To Configure Zone Files, refer to here .

Configure Keystone #2

Fri, 25 Oct 2019 00:00:00 +0000

Add Projects in Keystone.

This example is based on the emvironment like follows.

eth0|10.0.0.30 
+-----------+-----------+
| [ Control Node ] |
| |
| MariaDB RabbitMQ |
| Memcached httpd |
| Keystone |
+-----------------------+

Create and Load environment variables file.

The password for [OS_PASSWORD] is the one you set it on bootstrapping keystone

The URL for [OS_AUTH_URL] is the Keystone server’s hostname or IP address.

[root@dlp ~]# vi ~/keystonerc
export OS_PROJECT_DOMAIN_NAME=default export OS_USER_DOMAIN_NAME=default export OS_PROJECT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=adminpassword export OS_AUTH_URL=http://10.0.0.30:5000/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 export PS1='[\u@\h \W(keystone)]\$ '
[root@dlp ~]# chmod 600 ~/keystonerc
[root@dlp ~]# source ~/keystonerc
[root@dlp ~(keystone)]# echo "source ~/keystonerc " >> ~/.bash_profile

Create Projects.

Configure iSCSI Target

Mon, 14 Jul 2014 00:00:00 +0000

Configure Storage Server with iSCSI.

A storage on a network is called iSCSI Target, a Client which connects to iSCSI Target is called iSCSI Initiator. This example is based on the environment below.

+----------------------+ | +----------------------+
| [ iSCSI Target ] |10.0.0.30 | 10.0.0.31| [ iSCSI Initiator ] |
| dlp.srv.world +----------+----------+ www.srv.world |
| | | |
+----------------------+ +----------------------+

Install administration tools first.

[root@dlp ~]# yum -y install targetcli

Configure iSCSI Target.

For example, create an disk-image under the /var/lib/iscsi_disks directory and set it as a SCSI device.

Basic Operations

Sat, 12 Jul 2014 00:00:00 +0000

Basic Operarions with virsh command

Start Virtual Machine.

[root@dlp ~]# virsh start centos7
# Start Virtual Machine 'centos7'
Domain centos7 started
[root@dlp ~]# virsh start centos7 --console
# start and connect to console of 'centos7'
Domain centos7 started
Connected to domain centos7

Stop Virtual Machine.

[root@dlp ~]# virsh shutdown centos7
# Stop Virtual Machine 'centos7'
Domain centos7 is being shutdown
[root@dlp ~]# virsh destroy centos7
# Stop fourcely Virtual Machine 'centos7'
Domain centos7 destroyed

Set auto-start for Virtual Machines.

Password Authentication

Wed, 09 Jul 2014 00:00:00 +0000

Configure SSH Server to login to a server from remote computer.

OpenSSH is already installed by default even if you installed CentOS with [Minimal Install], so it’s not necessarry to install new packages. You can login with Password Authentication by default, but change some settings for security like follows.

[root@dlp ~]# vi /etc/ssh/sshd_config
# line 38: uncomment and change (prohibit root login remotely)
PermitRootLogin
no
[root@dlp ~]# systemctl restart sshd

If Firewalld is running

FireWall & SELinux

Tue, 08 Jul 2014 00:00:00 +0000

It’s possible to show Service Status of FireWall like follows. (enabled by default)

[root@dlp ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
 Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
 Active: active (running) since Thu 2015-03-31 19:36:27 JST; 2s ago
 Main PID: 1308 (firewalld)
 CGroup: /system.slice/firewalld.service
 └─1308 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
.....
.....

If you use FireWall service, it needs to modify settings of it because incoming requests for services are mostly not allowed by default. Refer to here to basic operation and settings to configure firewalld service .

Configure Zone Files

Wed, 24 Feb 2021 00:00:00 +0000

Configure Zone Files for each Zone set in [named.conf] .

Replace Network or Domain name on the example below to your own environment.

Create zone files that servers resolve IP address from Domain name.

The example below uses Internal network [10.0.0.0/24], Domain name [srv.world].

Replace to your own environment.

[root@dlp ~]# vi /var/named/srv.world.lan
$TTL 86400
@ IN SOA dlp.srv.world. root.srv.world. ( # any numerical values are OK for serial number but # recommendation is [YYYYMMDDnn] (update date + number) 2021022401 ;Serial
 3600 ;Refresh
 1800 ;Retry
 604800 ;Expire
 86400 ;Minimum TTL
) # define Name Server IN NS dlp.srv.world. # define Name Server's IP address IN A 10.0.0.30 # define Mail Exchanger Server IN MX 10 dlp.srv.world. # define each IP address of a hostname dlp IN A 10.0.0.30
www IN A 10.0.0.31

Create zone files that servers resolve Domain name from IP address.

Configure Glance

Fri, 25 Oct 2019 00:00:00 +0000

Install and Configure OpenStack Image Service (Glance).

This example is based on the emvironment like follows.

eth0|10.0.0.30 
+-----------+-----------+
| [ Control Node ] |
| |
| MariaDB RabbitMQ |
| Memcached httpd |
| Keystone Glance |
+-----------------------+

Add users and others for Glance in Keystone.

# add glance user (set in service project)
[root@dlp ~(keystone)]# openstack user create --domain default --project service --password servicepassword glance
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| default_project_id | 2b9c4eda462e40a4a3d03b0a38f12bf1 |
| domain_id | default |
| enabled | True |
| id | 50c2d1f8b5634bc9ac01ed90b76bdce6 |
| name | glance |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
# add glance user in admin role
[root@dlp ~(keystone)]# openstack role add --project service --user glance admin
# add service entry for glance
[root@dlp ~(keystone)]# openstack service create --name glance --description "OpenStack Image service" image
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Image service |
| enabled | True |
| id | 502a68e834764245bde8c9077b24dd09 |
| name | glance |
| type | image |
+-------------+----------------------------------+
# define Glance API node
[root@dlp ~(keystone)]# export controller=10.0.0.30
# add endpoint for glance (public)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne image public http://$controller:9292
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | e08fef5fd6934ef9b3f4b6ad93085fa6 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 502a68e834764245bde8c9077b24dd09 |
| service_name | glance |
| service_type | image |
| url | http://10.0.0.30:9292 |
+--------------+----------------------------------+
# add endpoint for glance (internal)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne image internal http://$controller:9292
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | d4488fa9630d44ffa560eeb462e251d0 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 502a68e834764245bde8c9077b24dd09 |
| service_name | glance |
| service_type | image |
| url | http://10.0.0.30:9292 |
+--------------+----------------------------------+
# add endpoint for glance (admin)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne image admin http://$controller:9292
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | 744f289f475940eeb5a742534905cb28 |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 502a68e834764245bde8c9077b24dd09 |
| service_name | glance |
| service_type | image |
| url | http://10.0.0.30:9292 |
+--------------+----------------------------------+

Add a User and Database on MariaDB for Glance.

Configure iSCSI Target (tgt)

Tue, 15 Dec 2015 00:00:00 +0000

Configure Storage Server with iSCSI.

This is the example of configuring iSCSI Target with scsi-target-utils.

Install scsi-target-utils.

# install from EPEL
[root@dlp ~]# yum --enablerepo=epel -y install scsi-target-utils

Configure iSCSI Target.

For example, create a disk image under the [/var/lib/iscsi_disks] directory and set it as a shared disk.

# create a disk image
[root@dlp ~]# mkdir /var/lib/iscsi_disks
[root@dlp ~]# dd if=/dev/zero of=/var/lib/iscsi_disks/disk01.img count=0 bs=1 seek=10G
[root@dlp ~]# vi /etc/tgt/targets.conf
# add follows to the end # if you set some devices, add <target>-</target> and set the same way with follows # naming rule : [ iqn.yaer-month.domain:any name ] <target iqn.2015-12.world.srv:target00> # provided devicce as a iSCSI target backing-store /var/lib/iscsi_disks/disk01.img # iSCSI Initiator's IP address you allow to connect initiator-address 10.0.0.31 # authentication info ( set anyone you like for "username", "password" ) incominguser username password </target>

If SELinux is enabled

Virtual Management Tools

Sat, 10 Jan 2015 00:00:00 +0000

Install useful tools for virt management.

Install virt tools.

[root@dlp ~]# yum -y install libguestfs-tools libguestfs-xfs virt-top

Get official OS image and Create a Virtual Machine.

(

If you’d like to do from OS installation, refer to here of [1]

)

# display available OS template
[root@dlp ~]# virt-builder -l
opensuse-13.1 x86_64 openSUSE 13.1
opensuse-13.2 x86_64 openSUSE 13.2
opensuse-42.1 x86_64 openSUSE Leap 42.1
opensuse-tumbleweed x86_64 openSUSE Tumbleweed
centos-6 x86_64 CentOS 6.6
centos-7.0 x86_64 CentOS 7.0
centos-7.1 x86_64 CentOS 7.1
centos-7.2 aarch64 CentOS 7.2 (aarch64)
centos-7.2 x86_64 CentOS 7.2
centos-7.3 x86_64 CentOS 7.3
centos-7.4 x86_64 CentOS 7.4
centos-7.5 x86_64 CentOS 7.5
.....
.....
# create an image of CentOS 7.5
[root@dlp ~]# export LIBGUESTFS_BACKEND=direct
[root@dlp ~]# virt-builder centos-7.5 --format qcow2 --size 20G -o centos75.qcow2 --root-password password
[ 3.8] Downloading: http://libguestfs.org/download/builder/centos-7.5.xz
[ 74.0] Planning how to build this image
[ 74.0] Uncompressing
.....
.....
 Output file: centos75.qcow2
 Output size: 20.0G
 Output format: qcow2
 Total usable space: 19.4G
 Free space: 18.3G (94%)
# to configure VM with the image above, run virt-install
[root@dlp ~]# virt-install \ --name centos-75 \ --ram 4096 \ --disk path=/var/kvm/images/centos75.qcow2 \ --vcpus 2 \ --os-type linux \ --os-variant rhel7.5 \ --network bridge=br0 \ --graphics none \ --serial pty \ --console pty \ --boot hd \ --import

“ls” a directory in a virtual machine.

Configure Networking

Wed, 07 Jan 2015 00:00:00 +0000

Set static IP address to the server.

Replace the interface name “eth0” for your own environment’s one.

# set hostname
[root@localhost ~]# hostnamectl set-hostname dlp.srv.world
# display devices
[root@localhost ~]# nmcli d
DEVICE TYPE STATE CONNECTION
eth0 ethernet connected eth0
lo loopback unmanaged --
# set IPv4 address ⇒ nmcli *** [IP address]
[root@localhost ~]# nmcli c modify eth0 ipv4.addresses 10.0.0.30/24
# set default gateway
[root@localhost ~]# nmcli c modify eth0 ipv4.gateway 10.0.0.1
# set DNS
[root@localhost ~]# nmcli c modify eth0 ipv4.dns 10.0.0.1
# set manual for static setting (it's "auto" for DHCP)
[root@localhost ~]# nmcli c modify eth0 ipv4.method manual
# restart the interface and reload the settings
[root@localhost ~]# nmcli c down eth0; nmcli c up eth0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/1)
# show settings
[root@localhost ~]# nmcli d show eth0
GENERAL.DEVICE: eth0
GENERAL.TYPE: ethernet
GENERAL.HWADDR: 00:0C:29:CD:9C:2D
GENERAL.MTU: 1500
GENERAL.STATE: 100 (connected)
GENERAL.CONNECTION: eth0
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/0
WIRED-PROPERTIES.CARRIER: on
IP4.ADDRESS[1]: ip = 10.0.0.30/24, gw = 10.0.0.1
IP4.DNS[1]: 10.0.0.1
IP6.ADDRESS[1]: ip = fe80::20c:29ff:fecd:9c2d/64, gw = ::
# show status
[root@localhost ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
 valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 link/ether 00:0c:29:cd:9c:2d brd ff:ff:ff:ff:ff:ff
 inet 10.0.0.30/24 brd 10.0.0.255 scope global eth0
 valid_lft forever preferred_lft forever
 inet6 fe80::20c:29ff:fecd:9c2d/64 scope link
 valid_lft forever preferred_lft forever

Disable IPv6 if you do not need it.

SSH File Transfer(CentOS)

Tue, 29 Jul 2014 00:00:00 +0000

It’s possible to transfer files with SSH.

The follows is for CentOS Clients, but if your Windows is Windows 10 Version 1803 like here , OpenSSH Client has been implemented as a Windows feature, so it’s possible to use scp, sftp commands like follows with the same usage on Windows command prompt, too.

It’s the example for using SCP (Secure Copy).

# command ⇒ scp [Option] Source Target
# copy the [test.txt] on local to remote server [www.srv.world]
[cent@dlp ~]$ scp ./test.txt cent@www.srv.world:~/
cent@www.srv.world's password:
# password of the user
test.txt 100% 10 0.0KB/s 00:00
# copy the [/home/fedora/test.txt] on remote server [www.srv.world] to the local
[cent@dlp ~]$ scp cent@www.srv.world:/home/cent/test.txt ./test.txt
cent@prox.srv.world's password:
test.txt 100% 10 0.0KB/s 00:00

It’s example to use SFTP (SSH File Transfer Protocol). SFTP server function is enabled by default but if not, enable it to add the line [Subsystem sftp /usr/libexec/openssh/sftp-server] in [/etc/ssh/sshd_config].

Verify Resolution

Wed, 24 Feb 2021 00:00:00 +0000

Start and Enable BIND.

[root@dlp ~]# systemctl enable --now named

If Firewalld is running

, allow DNS service. DNS uses [53/TCP,UDP].

[root@dlp ~]# firewall-cmd --add-service=dns --permanent
success
[root@dlp ~]# firewall-cmd --reload
success

Change DNS setting to refer to own DNS if need.

(replace [eth0] to your own environment).

[root@dlp ~]# nmcli connection modify eth0 ipv4.dns 10.0.0.30
[root@dlp ~]# nmcli connection down eth0; nmcli connection up eth0

Verify Name and Address Resolution. If [ANSWER SECTION] is shown, that’s OK.

Add VM Images

Fri, 25 Oct 2019 00:00:00 +0000

Add Virtual Machine images in Glance. For example on here, create and add CentOS 7 virtual machine image.

Install KVM HyperVisor, refer to here . It’s unnecessarry to set Bridge networking on the section [2] of the link.

Create CentOS 7 image on Glance Host.

# create a directory for disk image
[root@dlp ~(keystone)]# mkdir -p /var/kvm/images
# create a disk image
[root@dlp ~(keystone)]# qemu-img create -f qcow2 /var/kvm/images/centos7.img 10G
# install
[root@dlp ~(keystone)]# virt-install \ --name centos7 \ --ram 2048 \ --disk path=/var/kvm/images/centos7.img,format=qcow2 \ --vcpus 2 \ --os-type linux \ --os-variant rhel7 \ --graphics none \ --console pty,target_type=serial \ --location 'http://ftp.iij.ad.jp/pub/linux/centos/7/os/x86_64/' \ --extra-args 'console=ttyS0,115200n8 serial'
Starting install...
# installation starts

After installation, change settings in virtual machine like follows and shutdown it. * Delete or comment out the line [HWADDR] and [UUID] in [/etc/sysconfig/network-scripts/ifcfg-eth0] * Set [dhcp] for [BOOTPROTO] in [/etc/sysconfig/network-scripts/ifcfg-eth0] * Install [cloud-init] package and enable [cloud-init] service, and also add a user whose name is [centos]

Configure Services

Wed, 07 Jan 2015 00:00:00 +0000

It’s possible to make sure services’ status like follows.

# display the list of services which are running
[root@dlp ~]# systemctl -t service
UNIT LOAD ACTIVE SUB DESCRIPTION
auditd.service loaded active running Security Auditing Service
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
crond.service loaded active running Command Scheduler
dbus.service loaded active running D-Bus System Message Bus
getty@tty1.service loaded active running Getty on tty1
...
...
...
systemd-udevd.service loaded active running udev Kernel Device Manager
systemd-update-utmp.service loaded active exited Update UTMP about System Reboot/Shutdown
systemd-user-sessions.service loaded active exited Permit User Sessions
systemd-vconsole-setup.service loaded active exited Setup Virtual Console
tuned.service loaded active running Dynamic System Tuning Daemon

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.

39 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
# the list of all services
[root@dlp ~]# systemctl list-unit-files -t service
UNIT FILE STATE
auditd.service enabled
autovt@.service disabled
avahi-daemon.service enabled
blk-availability.service disabled
brandbot.service static
...
...
...
systemd-user-sessions.service static
systemd-vconsole-setup.service static
teamd@.service static
tuned.service enabled
wpa_supplicant.service disabled

125 unit files listed.

Stop and turn OFF auto-start setting for a service if you don’d need it. (it’s smartd as an example below)

Live Migration

Tue, 12 Aug 2014 00:00:00 +0000

This is the example to use Live Migration feature for Virtual Machines.

This requires 2 KVM host server and a storage server like follows. Configure DNS or hosts to resolve names or IP addresses normally, first.

+----------------------+
 | [ NFS Servver ] |
 | nfs.srv.world |
 | |
 +-----------+----------+
 |10.0.0.30
 |
+----------------------+ | +----------------------+
| [ KVM Host #1 ] |10.0.0.21 | 10.0.0.22| [ KVM Host #2 ] |
| +----------+----------+ |
| kvm01.srv.world | | kvm02.srv.world |
+----------------------+ +----------------------+

Configure Storage server which has virtual machine images. For Storage server, it’s OK to use NFS, iSCSI, GlusterFS and so on. This example uses NFS Storage server .

SSH File Transfer(Windows)

Tue, 29 Jul 2014 00:00:00 +0000

It’s possible to transfer files with SSH on Windows Client. This example shows to use WinSCP for it .

If your Windows is Windows 10 Version 1803 like here , OpenSSH Client has been implemented as a Windows feature, so it’s also possible to use scp, sftp commands for SSH File Transfer without WinSCP and others.

Install and start WinSCP, then following initial windows is displayed.

Input Hostname, Username, User’s Password and then Click [Advanced…] button.

Conf iSCSI Initiator

Mon, 14 Jul 2014 00:00:00 +0000

Configure iSCSI Initiator.

This example is based on the environment below.

+----------------------+ | +----------------------+
| [ iSCSI Target ] |10.0.0.30 | 10.0.0.31| [ iSCSI Initiator ] |
| dlp.srv.world +----------+----------+ www.srv.world |
| | | |
+----------------------+ +----------------------+

Configure iSCSI Initiator.

[root@www ~]# yum -y install iscsi-initiator-utils
[root@www ~]# vi /etc/iscsi/initiatorname.iscsi
# change to the same IQN you set on the iSCSI target server
InitiatorName=
iqn.2014-07.world.srv:www.srv.world
[root@www ~]# vi /etc/iscsi/iscsid.conf
# line 57: uncomment
node.session.auth.authmethod = CHAP
# line 61,62: uncomment and specify the username and password you set on the iSCSI target server
node.session.auth.username =
username
node.session.auth.password =
password
# discover target
[root@www ~]# iscsiadm -m discovery -t sendtargets -p 10.0.0.30
[ 635.510656] iscsi: registered transport (tcp)
10.0.0.30:3260,1 iqn.2014-07.world.srv:storage.target00
# confirm status after discovery
[root@www ~]# iscsiadm -m node -o show
# BEGIN RECORD 6.2.0.873-21
node.name = iqn.2014-07.world.srv:storage.target00
node.tpgt = 1
node.startup = automatic
node.leading_login = No
...
...
...
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD
# login to the target
[root@www ~]# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.2014-07.world.srv:storage.target00, portal: 10.0.0.30,3260] (multiple)
[ 708.383308] scsi2 : iSCSI Initiator over TCP/IP
[ 709.393277] scsi 2:0:0:0: Direct-Access LIO-ORG disk01 4.0 PQ: 0 ANSI: 5
[ 709.395709] scsi 2:0:0:0: alua: supports implicit and explicit TPGS
[ 709.398155] scsi 2:0:0:0: alua: port group 00 rel port 01
[ 709.399762] scsi 2:0:0:0: alua: port group 00 state A non-preferred supports TOlUSNA
[ 709.401763] scsi 2:0:0:0: alua: Attached
[ 709.402910] scsi 2:0:0:0: Attached scsi generic sg0 type 0
Login to [iface: default, target: iqn.2014-07.world.srv:storage.target00, portal: 10.0.0.30,3260] successful.
# confirm the established session
[root@www ~]# iscsiadm -m session -o show
tcp: [1] 10.0.0.30:3260,1 iqn.2014-07.world.srv:storage.target00 (non-flash)
# confirm the partitions
[root@www ~]# cat /proc/partitions
major minor #blocks name

 252 0 52428800 sda
 252 1 512000 sda1
 252 2 51915776 sda2
 253 0 4079616 dm-0
 253 1 47833088 dm-1
 8 0 20971520 sdb
# added new device provided from the target server as "sdb"

After setting iSCSI device, configure on Initiator to use it like follows.

Use View Statement

Wed, 24 Feb 2021 00:00:00 +0000

This is an example to use View Statement in [named.conf].

On this example, Configure both settings for Internal Network like here and settings for External Network like here with View Statement in [named.conf].

This example uses internal network [10.0.0.0/24], external network [172.16.0.80/29], domain name [srv.world], Replace them for your own environment.

( Actually, [172.16.0.80/29] is for private IP addresses, though. )

[root@dlp ~]# vi /etc/named.conf
.....
.....
# add : set ACL entry for local network
acl internal-network {
 10.0.0.0/24;
};

options {
 # change ( listen all )
 listen-on port 53 { any; };
 # change if need ( if not listen IPv6, set [none] )
 listen-on-v6 { any; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 secroots-file "/var/named/data/named.secroots";
 recursing-file "/var/named/data/named.recursing";
 # add local network set on [acl] section above
 # network range you allow to receive queries from hosts
 allow-query { localhost; internal-network; };
 # network range you allow to transfer zone files to clients
 # add secondary DNS servers if it exist
 allow-transfer { localhost; };

 .....
 .....

 recursion yes;

 dnssec-enable yes;
 dnssec-validation yes;

 managed-keys-directory "/var/named/dynamic";

 pid-file "/run/named/named.pid";
 session-keyfile "/run/named/session.key";

 /* https://fedoraproject.org/wiki/Changes/CryptoPolicy */
 include "/etc/crypto-policies/back-ends/bind.config";
};

logging {
 channel default_debug {
 file "data/named.run";
 severity dynamic;
 };
};

# change all lines follows
# set internal network zones
view "internal" {
 match-clients {
 localhost;
 internal-network;
 };
 zone "." IN {
 type hint;
 file "named.ca";
 };
 zone "srv.world" IN {
 type master;
 file "srv.world.lan";
 allow-update { none; };
 };
 zone "0.0.10.in-addr.arpa" IN {
 type master;
 file "0.0.10.db";
 allow-update { none; };
 };
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
};

# set external network zones
view "external" {
 # match all except targets defined on [match-clients] on internal section
 match-clients { any; };
 allow-query { any; };
 # not allow recursive queries
 recursion no;
 zone "srv.world" IN {
 type master;
 file "srv.world.wan";
 allow-update { none; };
 };
 zone "0.16.172.in-addr.arpa" IN {
 type master;
 file "0.16.172.db";
 allow-update { none; };
 };
};

For configuration of each Zone files set in [named.conf] above, refer to here .

Configure Nova #1

Fri, 25 Oct 2019 00:00:00 +0000

Install and Configure OpenStack Compute Service (Nova).

This example is based on the emvironment like follows.

eth0|10.0.0.30 
+-----------+-----------+
| [ Control Node ] |
| |
| MariaDB RabbitMQ |
| Memcached httpd |
| Keystone Glance |
+-----------------------+

Add users and others for Nova in Keystone.

# add nova user (set in service project)
[root@dlp ~(keystone)]# openstack user create --domain default --project service --password servicepassword nova
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| default_project_id | 2b9c4eda462e40a4a3d03b0a38f12bf1 |
| domain_id | default |
| enabled | True |
| id | ab07484ea050414abc6bf12be1081872 |
| name | nova |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
# add nova user in admin role
[root@dlp ~(keystone)]# openstack role add --project service --user nova admin
# add placement user (set in service project)
[root@dlp ~(keystone)]# openstack user create --domain default --project service --password servicepassword placement
+---------------------+----------------------------------+
| Field | Value |
+---------------------+----------------------------------+
| default_project_id | 2b9c4eda462e40a4a3d03b0a38f12bf1 |
| domain_id | default |
| enabled | True |
| id | fce4c43a6f55494d983abba2eaa26cd8 |
| name | placement |
| options | {} |
| password_expires_at | None |
+---------------------+----------------------------------+
# add placement user in admin role
[root@dlp ~(keystone)]# openstack role add --project service --user placement admin
# add service entry for nova
[root@dlp ~(keystone)]# openstack service create --name nova --description "OpenStack Compute service" compute
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | OpenStack Compute service |
| enabled | True |
| id | 88925f270ddd4cae9121903832de19ba |
| name | nova |
| type | compute |
+-------------+----------------------------------+
# add service entry for placement
[root@dlp ~(keystone)]# openstack service create --name placement --description "OpenStack Compute Placement service" placement
+-------------+-------------------------------------+
| Field | Value |
+-------------+-------------------------------------+
| description | OpenStack Compute Placement service |
| enabled | True |
| id | ef2201ae86a747548774d31b7871081b |
| name | placement |
| type | placement |
+-------------+-------------------------------------+
# define Nova API node
[root@dlp ~(keystone)]# export controller=10.0.0.30
# add endpoint for nova (public)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne compute public http://$controller:8774/v2.1/%\(tenant_id\)s
+--------------+------------------------------------------+
| Field | Value |
+--------------+------------------------------------------+
| enabled | True |
| id | 27859b28a81c4858873be443af82efe3 |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 88925f270ddd4cae9121903832de19ba |
| service_name | nova |
| service_type | compute |
| url | http://10.0.0.30:8774/v2.1/%(tenant_id)s |
+--------------+------------------------------------------+
# add endpoint for nova (internal)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne compute internal http://$controller:8774/v2.1/%\(tenant_id\)s
+--------------+------------------------------------------+
| Field | Value |
+--------------+------------------------------------------+
| enabled | True |
| id | 5822b95d290b4400b380b9e6c8979d04 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 88925f270ddd4cae9121903832de19ba |
| service_name | nova |
| service_type | compute |
| url | http://10.0.0.30:8774/v2.1/%(tenant_id)s |
+--------------+------------------------------------------+
# add endpoint for nova (admin)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne compute admin http://$controller:8774/v2.1/%\(tenant_id\)s
+--------------+------------------------------------------+
| Field | Value |
+--------------+------------------------------------------+
| enabled | True |
| id | 19d59af1988f416bab3455424551f75f |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | 88925f270ddd4cae9121903832de19ba |
| service_name | nova |
| service_type | compute |
| url | http://10.0.0.30:8774/v2.1/%(tenant_id)s |
+--------------+------------------------------------------+
# add endpoint for placement (public)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne placement public http://$controller:8778
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | 5d76ed498a4d47798aa706d2df0333cb |
| interface | public |
| region | RegionOne |
| region_id | RegionOne |
| service_id | ef2201ae86a747548774d31b7871081b |
| service_name | placement |
| service_type | placement |
| url | http://10.0.0.30:8778 |
+--------------+----------------------------------+
# add endpoint for placement (internal)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne placement internal http://$controller:8778
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | 039022c746404e079a2328a61d29b0b3 |
| interface | internal |
| region | RegionOne |
| region_id | RegionOne |
| service_id | ef2201ae86a747548774d31b7871081b |
| service_name | placement |
| service_type | placement |
| url | http://10.0.0.30:8778 |
+--------------+----------------------------------+
# add endpoint for placement (admin)
[root@dlp ~(keystone)]# openstack endpoint create --region RegionOne placement admin http://$controller:8778
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| enabled | True |
| id | 15e1eac702c741499615d42cab4f0609 |
| interface | admin |
| region | RegionOne |
| region_id | RegionOne |
| service_id | ef2201ae86a747548774d31b7871081b |
| service_name | placement |
| service_type | placement |
| url | http://10.0.0.30:8778 |
+--------------+----------------------------------+

Add a User and Database on MariaDB for Nova.

Configure Ceph Cluster #1

Wed, 12 Jun 2019 00:00:00 +0000

Install Distributed File System Ceph to Configure Storage Cluster.

For example on here, Configure Ceph Cluster with 3 Nodes like follows. Furthermore, each Storage Node has a free block device to use on Ceph Nodes. (use [/dev/sdb] on this example)

|
 +----------------------------+----------------------------+
 | | |
 |10.0.0.51 |10.0.0.52 |10.0.0.53 
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [node01.srv.world] | | [node02.srv.world] | | [node03.srv.world] |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| Manager Daemon | | | | |
+-----------------------+ +-----------------------+ +-----------------------+

Generate SSH key-pair on [Monitor Daemon] Node (call it Admin Node on here) and set it to each Node.

Storage Migration

Sat, 19 May 2018 00:00:00 +0000

This is the example to use Storage Migration function for Virtual Machines.

The different point of Storage Migration from Live Migration is that Storage Migration does not need a Storage server which has Virtual Machine images, when executing Storage Migration, Virtual Machine image on a KVM host is migrated on another KVM host like follows.

Storage Migration
 <------------------->
 
+----------------------+ +----------------------+
| [ KVM Host #1 ] |10.0.0.21 10.0.0.22| [ KVM Host #2 ] |
| +---------------------+ |
| kvm01.srv.world | | kvm02.srv.world |
+----------------------+ +----------------------+

Configure 2 KVM Host Server and Create a Virtual Machine on a KVM Host. 2 KVM Host Servers need to be resolv hostname or IP address each other via DNS or local hosts file.

SSH Keys Authentication

Tue, 29 Jul 2014 00:00:00 +0000

Configure SSH server to login with Key-Pair Authentication. Create a private key for client and a public key for server to do it.

Create Key-Pair for each user, so login with a common user on SSH Server Host and work like follows.

# create key pair
[cent@dlp ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/cent/.ssh/id_rsa):
# Enter if keep default
Created directory '/home/cent/.ssh'.
Enter passphrase (empty for no passphrase):
# set passphrase (set no passphrase to Enter with empty)
Enter same passphrase again:
Your identification has been saved in /home/cent/.ssh/id_rsa.
Your public key has been saved in /home/cent/.ssh/id_rsa.pub.
The key fingerprint is:
38:f1:b4:6d:d3:0e:59:c8:fa:1d:1d:48:86:f0:fe:74 cent@dlp.srv.world
The key's randomart image is:
[cent@dlp ~]$ mv ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

Transfer the secret key created on the Server to a Client, then it’s possible to login with keys authentication.

Update System

Tue, 08 Jul 2014 00:00:00 +0000

After installing CentOS, Update the system first if possible.

[root@dlp ~]# yum -y update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
Resolving Dependencies
--> Running transaction check
---> Package NetworkManager.x86_64 1:0.9.9.1-13.git20140326.4dba720.el7 will be updated
---> Package NetworkManager.x86_64 1:0.9.9.1-23.git20140326.4dba720.el7_0 will be an update
...
...
...
--> Finished Dependency Resolution

Dependencies Resolved

= == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == ==
 Package Arch Version Repository Size
= == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == ==
Installing:
 kernel x86_64 3.10.0-123.4.2.el7 updates 29 M
Updating:
 NetworkManager x86_64 1:0.9.9.1-23.git20140326... updates 1.4 M
 NetworkManager-glib x86_64 1:0.9.9.1-23.git20140326... updates 369 k
 NetworkManager-tui x86_64 1:0.9.9.1-23.git20140326... updates 190 k
 gnutls x86_64 3.1.18-9.el7_0 updates 609 k
 json-c x86_64 0.11-4.el7_0 updates 31 k
 kernel-tools x86_64 3.10.0-123.4.2.el7 updates 669 k
 kernel-tools-libs x86_64 3.10.0-123.4.2.el7 updates 609 k
 kexec-tools x86_64 2.0.4-32.el7.centos.1 updates 320 k
 libtasn1 x86_64 3.3-5.el7_0 updates 316 k
 mariadb-libs x86_64 1:5.5.37-1.el7_0 updates 752 k
 microcode_ctl x86_64 2:2.1-7.1.el7_0.1 updates 488 k
 openssl x86_64 1:1.0.1e-34.el7_0.3 updates 705 k
 openssl-libs x86_64 1:1.0.1e-34.el7_0.3 updates 939 k
 selinux-policy noarch 3.12.1-153.el7_0.10 updates 340 k
 selinux-policy-targeted noarch 3.12.1-153.el7_0.10 updates 3.8 M
 tuned noarch 2.3.0-11.el7_0.3 updates 145 k
 tzdata noarch 2014e-1.el7_0 updates 433 k

Transaction Summary
= == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == == ==
Install 1 Package
Upgrade 17 Packages

Total download size: 41 M
...
...
...
 Verifying : kernel-tools-libs-3.10.0-123.el7.x86_64 34/35
 Verifying : 1:NetworkManager-tui-0.9.9.1-13.git20140326.4dba720.el7.x86_64 35/35

Installed:
 kernel.x86_64 0:3.10.0-123.4.2.el7

Updated:
 NetworkManager.x86_64 1:0.9.9.1-23.git20140326.4dba720.el7_0 NetworkManager-glib.x86_64 1:0.9.9.1-2
 NetworkManager-tui.x86_64 1:0.9.9.1-23.git20140326.4dba720.el7_0 gnutls.x86_64 0:3.1.18-9.el7_0
 json-c.x86_64 0:0.11-4.el7_0 kernel-tools.x86_64 0:3.10.0-123.4.2.e
 kernel-tools-libs.x86_64 0:3.10.0-123.4.2.el7 kexec-tools.x86_64 0:2.0.4-32.el7.cent
 libtasn1.x86_64 0:3.3-5.el7_0 mariadb-libs.x86_64 1:5.5.37-1.el7_0
 microcode_ctl.x86_64 2:2.1-7.1.el7_0.1 openssl.x86_64 1:1.0.1e-34.el7_0.3
 openssl-libs.x86_64 1:1.0.1e-34.el7_0.3 selinux-policy.noarch 0:3.12.1-153.el7
 selinux-policy-targeted.noarch 0:3.12.1-153.el7_0.10 tuned.noarch 0:2.3.0-11.el7_0.3
 tzdata.noarch 0:2014e-1.el7_0

Complete!

摘自 Initial Settings : Update System · Server World，版权归原作者所有，此处仅作个人学习整理。

Set Alias (CNAME)

Wed, 24 Feb 2021 00:00:00 +0000

If you’d like to set Alias (Another Name) to a Host, set CNAME record in a zone file.

Set CNAME record in a zone file.

[root@dlp ~]# vi /var/named/srv.world.lan
$TTL 86400
@ IN SOA dlp.srv.world. root.srv.world. (
 # update serial if update zone file
 2021022402 ;Serial
 3600 ;Refresh
 1800 ;Retry
 604800 ;Expire
 86400 ;Minimum TTL
)
 IN NS dlp.srv.world.
 IN A 10.0.0.30
 IN MX 10 dlp.srv.world.

dlp IN A 10.0.0.30
www IN A 10.0.0.31

# [Alias] IN CNAME [Original Name]
ftp IN CNAME dlp.srv.world.
[root@dlp ~]# rndc reload
server reload successful
# verify resolution
[root@dlp ~]# dig ftp.srv.world.
; <<>> DiG 9.11.26-RedHat-9.11.26-3.el8 <<>> ftp.srv.world.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 921
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 95bb76ad3b5fba2e9d888d7e6035b959129ae8ee7a065547 (good)
;; QUESTION SECTION:
;ftp.srv.world. IN A

;; ANSWER SECTION:
ftp.srv.world. 86400 IN CNAME dlp.srv.world.
dlp.srv.world. 86400 IN A 10.0.0.30

;; AUTHORITY SECTION:
srv.world. 86400 IN NS dlp.srv.world.

;; Query time: 0 msec
;; SERVER: 10.0.0.30#53(10.0.0.30)
;; WHEN: Wed Feb 24 11:26:33 JST 2021
;; MSG SIZE rcvd: 118

摘自 BIND : Set Alias (CNAME) · Server World，版权归原作者所有，此处仅作个人学习整理。

Configure Ceph Cluster #2

Wed, 12 Jun 2019 00:00:00 +0000

Install Distributed File System Ceph to Configure Storage Cluster.

For example on here, Configure Ceph Cluster with 3 Nodes like follows. Furthermore, each Storage Node has a free block device to use on Ceph Nodes. (use [/dev/sdb] on this example)

|
 +----------------------------+----------------------------+
 | | |
 |10.0.0.51 |10.0.0.52 |10.0.0.53 
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [node01.srv.world] | | [node02.srv.world] | | [node03.srv.world] |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| Manager Daemon | | | | |
+-----------------------+ +-----------------------+ +-----------------------+

Configure [Monitor Daemon] and [Manager Daemon] first, refer to here .

Add Repositories

Mon, 22 Dec 2014 00:00:00 +0000

Add some useful external repositories to install useful softwares.

Install a plugin to add priorities to each installed repositories.

[root@dlp ~]# yum -y install yum-plugin-priorities
# set [priority=1] to official repository
[root@dlp ~]# sed -i -e "s/\]$/\]\npriority=1/g" /etc/yum.repos.d/CentOS-Base.repo

Add

EPEL

Repository which is provided from Fedora project.

[root@dlp ~]# yum -y install epel-release
# set [priority=5]
[root@dlp ~]# sed -i -e "s/\]$/\]\npriority=5/g" /etc/yum.repos.d/epel.repo
# for another way, change to [enabled=0] and use it only when needed
[root@dlp ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/epel.repo
# if [enabled=0], input a command to use the repository
[root@dlp ~]# yum --enablerepo=epel install [Package]

Add CentOS SCLo Software collections Repository.

SPICE Server

Sat, 12 Jul 2014 00:00:00 +0000

Install Desktop Virtualization “SPICE ( Simple Protocol for Independent Computing Environment )”. It’s possible to connect to virtual machines from remote client computer.

Install SPICE Server if it’s not installed yet. ( Normally it’s already installed with KVM for dependency )

[root@dlp ~]# yum -y install spice-server spice-protocol

Edit existing virtual machine’s xml-file and start virtual machine with SPICE like follows. This site’s example has created a virtual machine without graphics, so it’s OK to change settings like follows, but if you created virtual machine with a graphics, Remove “” and “” sections in xml file because qxl is used for graphics.

SFTP only + Chroot

Wed, 09 Jul 2014 00:00:00 +0000

Configure SFTP only + Chroot. Some users who are applied this settings can access only with SFTP and access to the permitted directories.

For example, Set /home as the Chroot directory.

# create a group for SFTP
[root@dlp ~]# groupadd sftp_users
# apply to a user [cent] for SFTP only
[root@dlp ~]# usermod -aG sftp_users cent
[root@dlp ~]# vi /etc/ssh/sshd_config
# line 132: comment out and add a line like below
#
Subsystem sftp /usr/libexec/openssh/sftp-server
Subsystem sftp internal-sftp
# add follows to the end
Match Group sftp_users X11Forwarding no AllowTcpForwarding no ChrootDirectory /home ForceCommand internal-sftp
[root@dlp ~]# systemctl restart sshd

Try to access with a user and make sure the settings.

Configure Chroot Environment

Wed, 24 Feb 2021 00:00:00 +0000

If you’d like to configure Chroot Environment for named, Set like follows.

After setting Chroot environment, configuration files are placed under [/var/named/chroot].

[named.conf] is placed under [/var/named/chroot/etc/named.conf], zone files are placed [/var/named/chroot/var/named/***].

When modify settings, Change them under [/var/named/chroot] files.

[root@dlp ~]# yum -y install bind-chroot
[root@dlp ~]# /usr/libexec/setup-named-chroot.sh /var/named/chroot on
[root@dlp ~]# systemctl disable --now named
[root@dlp ~]# systemctl enable --now named-chroot
Created symlink /etc/systemd/system/multi-user.target.wants/named-chroot.service → /usr/lib/systemd/system/named-chroot.service.
[root@dlp ~]# ll /var/named/chroot/etc
total 708
drwxr-x---. 3 root named 23 Feb 24 11:30 crypto-policies
-rw-r--r--. 2 root root 318 Oct 27 06:44 localtime
drwxr-x---. 2 root named 6 Feb 18 08:08 named
-rw-r-----. 1 root named 2373 Feb 24 11:24 named.conf
-rw-r-----. 1 root named 1029 Feb 18 08:08 named.rfc1912.zones
-rw-r--r--. 1 root named 1070 Feb 18 08:08 named.root.key
drwxr-x---. 3 root named 25 Feb 24 11:30 pki
-rw-r--r--. 1 root root 6568 Sep 10 2018 protocols
-rw-r-----. 1 root named 100 Feb 24 11:16 rndc.key
-rw-r--r--. 1 root root 692252 May 15 2020 services
[root@dlp ~]# ll /var/named/chroot/var/named
total 24
-rw-r--r--. 1 root root 313 Feb 24 11:14 0.0.10.db
drwxr-x---. 7 root named 61 Feb 24 11:30 chroot
drwxrwx---. 2 named named 23 Feb 24 11:16 data
drwxrwx---. 2 named named 108 Feb 24 11:30 dynamic
-rw-r-----. 1 root named 2253 Feb 18 08:08 named.ca
-rw-r-----. 1 root named 152 Feb 18 08:08 named.empty
-rw-r-----. 1 root named 152 Feb 18 08:08 named.localhost
-rw-r-----. 1 root named 168 Feb 18 08:08 named.loopback
drwxrwx---. 2 named named 6 Feb 18 08:07 slaves
-rw-r--r--. 1 root root 404 Feb 24 11:26 srv.world.lan

摘自 BIND : Configure Chroot Environment · Server World，版权归原作者所有，此处仅作个人学习整理。

Use Block Device

Wed, 12 Jun 2019 00:00:00 +0000

Configure a Client Host [dlp] to use Ceph Storage like follows.

|
 +--------------------+ |
 | [dlp.srv.world] |10.0.0.30 |
 | Ceph Client +-----------+
 | | |
 +--------------------+ |
 +----------------------------+----------------------------+
 | | |
 |10.0.0.51 |10.0.0.52 |10.0.0.53 
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [node01.srv.world] | | [node02.srv.world] | | [node03.srv.world] |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| Manager Daemon | | | | |
+-----------------------+ +-----------------------+ +-----------------------+

For example, Create a block device and mount it on a Client Host.

Transfer SSH public key to Client Host and Configure it from Admin Node.

SPICE Client

Sat, 12 Jul 2014 00:00:00 +0000

Connect to a virtual machine that is running with SPICE. This example uses CentOS 7 for SPICE client.

Start [Remote Desktop Viewer] like follows.

Click [Connect] button.

Select [SPICE] for [Protocol] field and input [(server name or IP address):(port)] for [Host] field and then click [Connect] button.

Input the password you set and click [Authenticate] button.

After successing authentication, it’s possible to connect like follows.

KVM : SPICE Client : Windows

SPICE client for Windows is also provided, so it’s possible to connect from Windows. This example shows to use SPICE client on Windows 10.

SSH Port Forwarding

Wed, 09 Jul 2014 00:00:00 +0000

It’s possible to forward a port to another port with SSH port forwarding.

For example, configure the forwarding settings that the 8081 on the local is forwarded to the 5901(VNC) on the local. This example shows simply setting but it’s possible to forward most ports to most ports on the local or on other servers.

Start a VNC process on display #1 with a common user

, next, Configure like follows.

Configure vim

Tue, 08 Jul 2014 00:00:00 +0000

Install and Configure vim that is more convenience than vi.

Install vim

[root@dlp ~]# yum -y install vim-enhanced

Set command alias. ( Apply to all users below. If you apply to a user, Write the same settings in ‘~/.bashrc’ )

[root@dlp ~]# vi /etc/profile
# add at the last line
alias vi='vim'
[root@dlp ~]# source /etc/profile
# reload

Configure vim. ( Apply to a user below. If you applly to all users, Write the same settings in ‘/etc/vimrc’, some settings are applied by default though. )

Configure Secondary Server

Wed, 24 Feb 2021 00:00:00 +0000

Configure DNS Secondary Server (Slave Server).

On this example, it shows to configure DNS Secondary Server [ns.server.education] (192.168.100.85) that DNS Master Server is [dlp.srv.world] (172.16.0.82) configured like here . Replace IP address and Hostname to your own environment.

Configure on DNS Master Server Host.

[root@dlp ~]# vi /etc/named.conf
.....
.....
options {
 listen-on port 53 { any; };
 listen-on-v6 { any; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 secroots-file "/var/named/data/named.secroots";
 recursing-file "/var/named/data/named.recursing";
 allow-query { localhost; internal-network; };
 # add secondary server to allow to transfer zone files
 allow-transfer { localhost; 192.168.100.85; };
.....
.....
[root@dlp ~]# vi /var/named/srv.world.wan
$TTL 86400
@ IN SOA dlp.srv.world. root.srv.world. (
 # update serial if update zone file
 2021022403 ;Serial
 3600 ;Refresh
 1800 ;Retry
 604800 ;Expire
 86400 ;Minimum TTL
)
 IN NS dlp.srv.world.
 # add secondary server
 IN NS ns.server.education.
 IN A 172.16.0.82
 IN MX 10 dlp.srv.world.

dlp IN A 172.16.0.82
www IN A 172.16.0.83
[root@dlp ~]# systemctl restart named

Configure on DNS Secondary Server Host.

Use File System

Wed, 12 Jun 2019 00:00:00 +0000

Configure a Client Host [dlp] to use Ceph Storage like follows.

|
 +--------------------+ |
 | [dlp.srv.world] |10.0.0.30 |
 | Ceph Client +-----------+
 | | |
 +--------------------+ |
 +----------------------------+----------------------------+
 | | |
 |10.0.0.51 |10.0.0.52 |10.0.0.53 
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [node01.srv.world] | | [node02.srv.world] | | [node03.srv.world] |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| Manager Daemon | | | | |
+-----------------------+ +-----------------------+ +-----------------------+

For example, mount as Filesystem on a Client Host.

Transfer SSH public key to Client Host and Configure it from Admin Node.

SSH X11 Forwarding

Wed, 15 Apr 2015 00:00:00 +0000

It’s possible to display and use GUI applications in remote server on local client with SSH X11 Forwarding.

Enable X11 Forwarding on SSH server.

[root@dlp ~]# vi /etc/ssh/sshd_config
# line 102: uncomment
X11Forwarding yes
X11DisplayOffset 10
[root@dlp ~]# systemctl restart sshd

How to use on a Client : CentOS

This is the way to use GUI App on a CentOS client which desktop environment is installed .

Start Terminal and connect to the SSH server which is enabled X11 Forwarding with “ssh -XC xxx” like follows.

Nested KVM

Sat, 12 Jul 2014 00:00:00 +0000

Configure nested KVM. It’s possible to install KVM and create virtual machines as nested KVM on KVM host.

Enable the setting for Nested KVM.

[root@dlp ~]# vi /etc/modprobe.d/kvm-nested.conf
# create new
options kvm_intel nested=1
[root@dlp ~]# modprobe -r kvm_intel
# unload
[root@dlp ~]# modprobe kvm_intel
# reload again
[root@dlp ~]# cat /sys/module/kvm_intel/parameters/nested
Y
# just enabled

Edit the configuration of a virtual machine you’d like to set nested like follows.

It’s OK to configure nested KVM and you can create virtual machines on the virtual machine nested.

Configure sudo

Tue, 08 Jul 2014 00:00:00 +0000

Configure sudo to separate users’ duty if some people share privileges.

It’s unnecessarry to install sudo manually because it is installed by default even if “Minimal Install”.

Transfer root privilege to a user all.

[root@dlp ~]# visudo
# add at the last line: user 'cent' can use all root privilege
cent ALL=(ALL) ALL
# how to write ⇒ destination host=(owner) command
# make sure with the user 'cent'
[cent@dlp ~]$ /usr/bin/ cat /etc/shadow
cat: /etc/shadow: Permission denied
# denied normally
[cent@dlp ~]$ sudo /usr/bin/ cat /etc/shadow
[sudo] password for cent:
# own password
daemon:*:16231:0:99999:7:::
adm:*:16231:0:99999:7:::
lp:*:16231:0:99999:7:::
...
...
# just executed

In addition to the setting [1], set that some commands are not allowed.

Ceph Object Gateway

Wed, 12 Jun 2019 00:00:00 +0000

Enable Ceph Object Gateway (RADOSGW) to access to Ceph Cluster Storage via Amazon S3 or OpenStack Swift compatible API. This example is based on the environment like follows.

|
 +--------------------+ | +----------------------+
 | [dlp.srv.world] |10.0.0.30 | 10.0.0.31| [www.srv.world] |
 | Ceph Client +-----------+-----------+ RADOSGW |
 | | | | |
 +--------------------+ | +----------------------+
 +----------------------------+----------------------------+
 | | |
 |10.0.0.51 |10.0.0.52 |10.0.0.53 
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [node01.srv.world] | | [node02.srv.world] | | [node03.srv.world] |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| Manager Daemon | | | | |
+-----------------------+ +-----------------------+ +-----------------------+

Transfer required files to RADOSGW Node and Configure it from Admin Node.

GPU Passthrough

Wed, 31 Jan 2018 00:00:00 +0000

Configure GPU Passthrough for Virtual Machines.

By this configuration, it’s possible to use GPU on Virtual Machines and run GPU Computing by CUDA , Machine learning/Deep Learning by TensorFlow .

Before configuration, Enable VT-d (Intel) or AMD IOMMU (AMD) on BIOS Setting first.

Enable IOMMU on KVM Host.

[root@dlp ~]# vi /etc/default/grub
# line 6: add (if AMD CPU, add [amd_iommu=on])
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet
intel_iommu=on
"
GRUB_DISABLE_RECOVERY="true"
[root@dlp ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
[root@dlp ~]# reboot
# after rebooting, verify IOMMU is enabled like follows
[root@dlp ~]# dmesg | grep -E "DMAR|IOMMU"
[ 0.000000] ACPI: DMAR 00000000bf79e0d0 00118 (v01 AMI OEMDMAR 00000001 MSFT 00000097)
[ 0.000000] DMAR: IOMMU enabled
[ 0.069846] DMAR: Host address width 40
[ 0.069848] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[ 0.069856] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[ 0.069858] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[ 0.069859] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
[ 0.069861] DMAR: ATSR flags: 0x0
[ 0.069863] DMAR-IR: IOAPIC id 6 under DRHD base 0xfbffe000 IOMMU 0
[ 0.070128] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.960840] DMAR: dmar0: Using Queued invalidation
[ 0.960860] DMAR: Setting RMRR:
[ 0.960890] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff]
[ 0.960925] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff]
[ 0.960958] DMAR: Setting identity map for device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff]
[ 0.960994] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff]
[ 0.961031] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff]
[ 0.961064] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff]
[ 0.961100] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff]
[ 0.961132] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff]
[ 0.961153] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff]
[ 0.961167] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff]
[ 0.961182] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff]
[ 0.961196] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff]
[ 0.961210] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff]
[ 0.961224] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff]
[ 0.961238] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff]
[ 0.961255] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff]
[ 0.961270] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.961287] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.961424] DMAR: Intel(R) Virtualization Technology for Directed I/O

Enable vfio-pci kernel module. It’s OK to keep common kernel provided by RHEL/CentOS official.

Use SSHPass

Wed, 15 Apr 2015 00:00:00 +0000

Use SSHPass to automate inputting password on password authentication.

This is convenient but has security risks (leak of password), take special care if you use it.

Install SSHPass.

# install from EPEL
[root@dlp ~]# yum --enablerepo=epel -y install sshpass

How to use SSHPass.

# -p password : from argument
[cent@dlp ~]$ sshpass -p password ssh 10.0.0.51 hostname
node01.srv.world
# -f file : from file
[cent@dlp ~]$ echo 'password' > sshpass.txt
[cent@dlp ~]$ chmod 600 sshpass.txt
[cent@dlp ~]$ sshpass -f sshpass.txt ssh 10.0.0.51 hostname
node01.srv.world
# -e : from env variable
[cent@dlp ~]$ export SSHPASS=password
[cent@dlp ~]$ sshpass -e ssh 10.0.0.51 hostname
node01.srv.world

摘自 Use SSHPass · Server World，版权归原作者所有，此处仅作个人学习整理。

Configure DHCP Server

Fri, 11 Jul 2014 00:00:00 +0000

Configure DHCP ( Dynamic Host Configuration Protocol ) Server.

Install and configure DHCP Server.

[root@dlp ~]# yum -y install dhcp
[root@dlp ~]# vi /etc/dhcp/dhcpd.conf
# create new
# specify domain name option domain-name     "srv.world"; # specify name server's hostname or IP address option domain-name-servers     dlp.srv.world; # default lease time default-lease-time 600; # max lease time max-lease-time 7200; # this DHCP server to be declared valid authoritative; # specify network address and subnet mask subnet 10.0.0.0 netmask 255.255.255.0 { # specify the range of lease IP address range dynamic-bootp 10.0.0.200 10.0.0.254; # specify broadcast address option broadcast-address 10.0.0.255; # specify default gateway option routers 10.0.0.1; }
[root@dlp ~]# systemctl start dhcpd
[root@dlp ~]# systemctl enable dhcpd

If Firewalld is running, allow DHCP service. DHCP Server uses 67/UDP.

Cron's Setting

Tue, 08 Jul 2014 00:00:00 +0000

The regular jobs are controled by anacron by default. But anacron runs randomly in a day(at 3-22), so if you’d like to use cron and control jobs at a time, change like follows.

But Consider to change it if your server is virtualization environment. In virtual server, some virtual machines are running and if they executes regular jobs at a time all, the server will have many loading at a time. So it’s necessary to change time on each machine by hand, or Keep anacron setting.

Enable Dashboard

Wed, 12 Jun 2019 00:00:00 +0000

Enable Ceph Dashboard to manage Ceph Cluster on Web Console. This example is based on the environment like follows.

|
 +--------------------+ | +----------------------+
 | [dlp.srv.world] |10.0.0.30 | 10.0.0.31| [www.srv.world] |
 | Ceph Client +-----------+-----------+ RADOSGW |
 | | | | |
 +--------------------+ | +----------------------+
 +----------------------------+----------------------------+
 | | |
 |10.0.0.51 |10.0.0.52 |10.0.0.53 
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [node01.srv.world] | | [node02.srv.world] | | [node03.srv.world] |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| Manager Daemon | | | | |
+-----------------------+ +-----------------------+ +-----------------------+

Enable Dashboard module on [Manager Daemon] Node.

Furthermore, Dashboard requires SSL/TLS. Create a self-signed certificate on this example.

UEFI Boot

Sat, 03 Feb 2018 00:00:00 +0000

Boot Virtual Machines with using UEFI (Unified Extensible Firmware Interface).

Install OVMF (Open Virtual Machine Firmware).

[root@dlp ~]# vi /etc/yum.repos.d/kraxel.repo
# create new
[qemu-firmware-jenkins] name=firmware for qemu, built by jenkins, fresh from git repos baseurl=https://www.kraxel.org/repos/jenkins/ enabled=0 gpgcheck=0
[root@dlp ~]# yum --enablerepo=qemu-firmware-jenkins -y install OVMF
[root@dlp ~]# vi /etc/libvirt/qemu.conf
# line 682: add nvram = [
 "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd",
]
[root@dlp ~]# systemctl restart libvirtd

Update QEMU.

[root@dlp ~]# /usr/libexec/qemu-kvm -version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-141.el7_4.6), Copyright (c) 2003-2008 Fabrice Bellard
[root@dlp ~]# yum -y install centos-release-qemu-ev
# disable usually
[root@dlp ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo
# for this installing, [qemu-kvm] package is replaced to [qemu-kvm-ev] package
[root@dlp ~]# yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
[root@dlp ~]# systemctl restart libvirtd
[root@dlp ~]# /usr/libexec/qemu-kvm -version
QEMU emulator version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

It’s OK all, Create a VM to install Windows Server 2016 as an example.

Use SSH-Agent

Wed, 15 Apr 2015 00:00:00 +0000

Use SSH-Agent to automate inputting passphrase on key-pair authentication.

It’s necessarry to set key-pair first .

How to use SSH-Agent.

# start SSH-Agent
[cent@dlp ~]$ eval `ssh-agent`
Agent pid 2168
# add Identity
[cent@dlp ~]$ ssh-add
Enter passphrase for /home/cent/.ssh/id_rsa:
Identity added: /home/cent/.ssh/id_rsa (/home/cent/.ssh/id_rsa)
# confirm
[cent@dlp ~]$ ssh-add -l
2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:4c:b2 /home/cent/.ssh/id_rsa (RSA)
# try to connect with SSH without passphrase
[cent@dlp ~]$ ssh node01.srv.world hostname
node01.srv.world
# exit from SSH-Agent
[cent@dlp ~]$ eval `ssh-agent -k`
Agent pid 2168 killed

摘自 Use SSH-Agent · Server World，版权归原作者所有，此处仅作个人学习整理。

Configure DHCP Client

Fri, 11 Jul 2014 00:00:00 +0000

Configure DHCP Client. DHCP Client uses 68/UDP.

For CentOS Clients, Configure like follows. (Replace the section ‘ifcfg-***’ to your own interface name)

[root@client ~]# nmcli c modify ens3 ipv4.method auto
[root@client ~]# nmcli c down ens3; nmcli c up ens3

Configure DHCP Client : Windows

It’s the example for configuring on Windows 10 Client.

Open the network settings and select “internet protocol” and click “property” button.

Check boxes like follows, then IP address is re-configured automatically.

CephFS + NFS-Ganesha

Wed, 12 Jun 2019 00:00:00 +0000

Install NFS-Ganesha to mount Ceph File System with NFS protocol.

For example, Configure NFS Export setting to CephFS like here .

Install and Configure NFS-Ganesha on CephFS Node.

[root@node01 ~]# yum -y install centos-release-nfs-ganesha30
[root@node01 ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-NFS-Ganesha-30.repo
[root@node01 ~]# yum --enablerepo=centos-nfs-ganesha30 -y install nfs-ganesha-ceph
[root@node01 ~]# mv /etc/ganesha/ganesha.conf /etc/ganesha/ganesha.conf.org
[root@node01 ~]# vi /etc/ganesha/ganesha.conf
# create new NFS_CORE_PARAM { # disable NLM Enable_NLM = false; # disable RQUOTA (not suported on CephFS) Enable_RQUOTA = false; # NFS protocol Protocols = 4;
}
EXPORT_DEFAULTS { # default access mode Access_Type = RW;
}
EXPORT { # uniq ID Export_Id = 101; # mount path of CephFS Path = "/";
 FSAL {
 name = CEPH; # hostname or IP address of this Node hostname="10.0.0.51";
 } # setting for root Squash Squash="No_root_squash"; # NFSv4 Pseudo path Pseudo="/vfs_ceph"; # allowed security options SecType = "sys";
}
LOG { # default log level Default_Log_Level = WARN;
}
[root@node01 ~]# systemctl enable --now nfs-ganesha

If SELinux is enabled

VMWare ESXi on KVM

Sat, 03 Feb 2018 00:00:00 +0000

Install VMWare ESXi on KVM Virtual Machine.

Enable Nested KVM, refer to [1] of here .

Update QEMU.

[root@dlp ~]# /usr/libexec/qemu-kvm -version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-141.el7_4.6), Copyright (c) 2003-2008 Fabrice Bellard
[root@dlp ~]# yum -y install centos-release-qemu-ev
# disable usually
[root@dlp ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo
# for this installing, [qemu-kvm] package is replaced to [qemu-kvm-ev] package
[root@dlp ~]# yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
[root@dlp ~]# systemctl restart libvirtd
[root@dlp ~]# /usr/libexec/qemu-kvm -version
QEMU emulator version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

It’s OK all, Install VMWare ESXi 6.5 as an example.

Use Parallel SSH

Wed, 15 Apr 2015 00:00:00 +0000

Install Parallel SSH to connect to multiple hosts.

Install PSSH.

# install from EPEL
[root@dlp ~]# yum --enablerepo=epel -y install pssh

How to use PSSH.

This is the case for key-pair authentication without passphrase.

If passphrase is set in key-pair,

start SSH-Agent first to automate inputting passphrase

# connect to hosts and execute hostname command
[cent@dlp ~]$ pssh -H "10.0.0.51 10.0.0.52" -i "hostname"
[1] 17:28:02 [SUCCESS] 10.0.0.51
node01.srv.world
[2] 17:28:02 [SUCCESS] 10.0.0.52
node02.srv.world
# it's possible to read host list fron a file
[cent@dlp ~]$ vi pssh_hosts.txt
# write hosts per line like follows cent@10.0.0.51 cent@10.0.0.52
[cent@dlp ~]$ pssh -h pssh_hosts.txt -i "uptime"
[1] 19:37:59 [SUCCESS] cent@10.0.0.52
 19:37:59 up 1:35, 0 users, load average: 0.00, 0.00, 0.00
[2] 19:37:59 [SUCCESS] cent@10.0.0.51
 19:37:59 up 1:35, 0 users, load average: 0.00, 0.00, 0.00

It’s possible to connect with password authentication too, but it needs passwords on all hosts are the same one.

Add or Remove OSDs

Wed, 12 Jun 2019 00:00:00 +0000

This is how to add or remove OSDs from exisiting Cluster.

|
 +--------------------+ |
 | [dlp.srv.world] |10.0.0.30 |
 | Ceph Client +-----------+
 | | |
 +--------------------+ |
 +----------------------------+----------------------------+
 | | |
 |10.0.0.51 |10.0.0.52 |10.0.0.53 
+-----------+-----------+ +-----------+-----------+ +-----------+-----------+
| [node01.srv.world] | | [node02.srv.world] | | [node03.srv.world] |
| Object Storage +----+ Object Storage +----+ Object Storage |
| Monitor Daemon | | | | |
| Manager Daemon | | | | |
+-----------------------+ +-----------------------+ +-----------------------+

For example, Add a [node04] node to OSDs on Admin Node.

For Block device on new [node04] Node, use [/dev/sdb] on this example.

Configure Administration Server

Fri, 12 Oct 2018 00:00:00 +0000

Install oVirt which is the management tool for Virtualization Environment and also it the OSS edition of Red Hat Enterprise Virtualization. This example is based on the environment like follows.

+----------------------+ | +----------------------+
| [ oVirt Management ] |10.0.0.30 | 10.0.0.51| [ oVirt Node ] |
| dlp.srv.world +----------+----------+ node01.srv.world |
| | | |
+----------------------+ +----------------------+

Configure oVirt Administration Server.

[root@dlp ~]# yum -y install http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm
[root@dlp ~]# yum -y install ovirt-engine
[root@dlp ~]# engine-setup
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
 Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf']
 Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20181012101318-1y4czu.log
 Version: otopi-1.7.8 (otopi-1.7.8-1.el7)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment setup
[ INFO ] Stage: Environment customization

 --== PRODUCT OPTIONS ==--

 # Enter with default all
 Configure Engine on this host (Yes, No) [Yes]:
 Configure ovirt-provider-ovn (Yes, No) [Yes]:
 Configure Image I/O Proxy on this host (Yes, No) [Yes]:
 Configure WebSocket Proxy on this host (Yes, No) [Yes]:

 * Please note * : Data Warehouse is required for the engine.
 If you choose to not configure it on this host, you have to configure
 it on a remote host, and then configure the engine on this host so
 that it can access the database of the remote Data Warehouse host.
 # Enter with default all
 Configure Data Warehouse on this host (Yes, No) [Yes]:
 Configure VM Console Proxy on this host (Yes, No) [Yes]:

 --== PACKAGES ==--

[ INFO ] Checking for product updates...
[ INFO ] No product updates found

 --== NETWORK CONFIGURATION ==--

 # make sure your Hostname
 Host fully qualified DNS name of this server [dlp.srv.world]:
 Setup can automatically configure the firewall on this system.
 Note: automatic configuration of the firewall may overwrite current settings.
 NOTICE: iptables is deprecated and will be removed in future releases
 # Enter with default
 Do you want Setup to configure the firewall? (Yes, No) [Yes]:
[ INFO ] firewalld will be configured as firewall manager.

 --== DATABASE CONFIGURATION ==--

 # select it for Data Warehouse DB (choose Local on this example)
 Where is the DWH database located? (Local, Remote) [Local]:
 Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications.
 # select it for Data Warehouse DB Config (choose Automatic on this example)
 Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:
 # select it for Engine DB (choose Local on this example)
 Where is the Engine database located? (Local, Remote) [Local]:
 Setup can configure the local postgresql server automatically for the engine to run. This may conflict with existing applications.
 # select it for Engine DB Config (choose Automatic on this example)
 Would you like Setup to automatically configure postgresql and create Engine database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:

 --== OVIRT ENGINE CONFIGURATION ==--

 # set oVirt admin password
 Engine admin password:
 Confirm engine admin password:
 # select application mode (choose Both on this example)
 Application mode (Virt, Gluster, Both) [Both]:
 # select it to use default credentials or not
 Use default credentials (admin@internal) for ovirt-provider-ovn (Yes, No) [Yes]:

 --== STORAGE CONFIGURATION ==--

 # select it when removing VM disk, also remove blocks for VM disk or not (choose No on this example)
 Default SAN wipe after delete (Yes, No) [No]:

 --== PKI CONFIGURATION ==--

 # specify Organization name for certificate
 Organization name for certificate [srv.world]:

 --== APACHE CONFIGURATION ==--

 Setup can configure the default page of the web server to present the application home page. This may conflict with existing applications.
 # select it to set default page of apprication for Web server or not (choose Yes on this example)
 Do you wish to set the application as the default page of the web server? (Yes, No) [Yes]:
 Setup can configure apache to use SSL using a certificate issued from the internal CA.
 # select it for Certificates (choose Automatic on this example)
 Do you wish Setup to configure that, or prefer to perform that manually? (Automatic, Manual) [Automatic]:

 --== SYSTEM CONFIGURATION ==--

 --== MISC CONFIGURATION ==--

 # select Data Warehouse Sampling Scale
 # if Data Warehouse is on remote, recommended [Full]
 # if Data Warehouse in on local, recommended [Basic]
 Please choose Data Warehouse sampling scale:
 (1) Basic
 (2) Full
 (1, 2)[1]: 1

 --== END OF CONFIGURATION ==--

[ INFO ] Stage: Setup validation

 --== CONFIGURATION PREVIEW ==--

 Application mode : both
 Default SAN wipe after delete : False
 Firewall manager : firewalld
 Update Firewall : True
 Host FQDN : dlp.srv.world
 Configure local Engine database : True
 Set application as default page : True
 Configure Apache SSL : True
 Engine database secured connection : False
 Engine database user name : engine
 Engine database name : engine
 Engine database host : localhost
 Engine database port : 5432
 Engine database host name validation : False
 Engine installation : True
 PKI organization : srv.world
 Set up ovirt-provider-ovn : True
 Configure WebSocket Proxy : True
 DWH installation : True
 DWH database host : localhost
 DWH database port : 5432
 Configure local DWH database : True
 Configure Image I/O Proxy : True
 Configure VMConsole Proxy : True

 # make sure selection and [OK] to proceed
 Please confirm installation settings (OK, Cancel) [OK]:
[ INFO ] Stage: Transaction setup
[ INFO ] Stopping engine service
[ INFO ] Stopping ovirt-fence-kdump-listener service
[ INFO ] Stopping dwh service
[ INFO ] Stopping Image I/O Proxy service
[ INFO ] Stopping vmconsole-proxy service
[ INFO ] Stopping websocket-proxy service
[ INFO ] Stage: Misc configuration
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Upgrading CA
[ INFO ] Initializing PostgreSQL
[ INFO ] Creating PostgreSQL 'engine' database
[ INFO ] Configuring PostgreSQL
[ INFO ] Creating PostgreSQL 'ovirt_engine_history' database
[ INFO ] Configuring PostgreSQL
[ INFO ] Creating CA
[ INFO ] Creating/refreshing DWH database schema
[ INFO ] Configuring Image I/O Proxy
[ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts
[ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts
[ INFO ] Configuring WebSocket Proxy
[ INFO ] Creating/refreshing Engine database schema
[ INFO ] Creating/refreshing Engine 'internal' domain database schema
[ INFO ] Adding default OVN provider to database
[ INFO ] Adding OVN provider secret to database
[ INFO ] Setting a password for internal user admin
[ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Starting engine service
[ INFO ] Starting dwh service
[ INFO ] Restarting ovirt-vmconsole proxy service

 --== SUMMARY ==--

[ INFO ] Restarting httpd
 Please use the user 'admin@internal' and password specified in order to login
 Web access is enabled at:
 http://dlp.srv.world:80/ovirt-engine
 https://dlp.srv.world:443/ovirt-engine
 Internal CA 3F:07:3B:94:3B:38:77:3F:49:71:FD:44:22:1F:24:9E:2F:BC:26:EB
 SSH fingerprint: SHA256:ZlvFsKtarAQlWGoUvcNDQ309KvD7AHaaxWIxd8ZpVyc

 --== END OF SUMMARY ==--

[ INFO ] Stage: Clean up
 Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20181012101318-1y4czu.log
[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20181012102044-setup.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of setup completed successfully

摘自 oVirt 4.2 : Configure oVirt Administration Server · Server World，版权归原作者所有，此处仅作个人学习整理。

Install GlusterFS

Tue, 25 May 2021 00:00:00 +0000

Install GlusterFS to Configure Storage Cluster.

It is strongly recommended to use partitions for GlusterFS volumes that are different from the / partition.

On this example, it is based on the environment that all nodes has [sdb1] and mount it to [/glusterfs].

Install GlusterFS Server on all Nodes in Cluster.

[root@node01 ~]# yum -y install centos-release-gluster9
[root@node01 ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-Gluster-9.repo
[root@node01 ~]# yum --enablerepo=centos-gluster9 -y install glusterfs-server
[root@node01 ~]# systemctl enable --now glusterd
[root@node01 ~]# gluster --version
glusterfs 9.2
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.

If Firewalld is running

Add oVirt Compute Nodes

Fri, 12 Oct 2018 00:00:00 +0000

Add oVirt Compute Nodes on oVirt Web Admin Console from any Client Hosts. This example is based on the environment like follows.

+----------------------+ | +----------------------+
| [ oVirt Management ] |10.0.0.30 | 10.0.0.51| [ oVirt Node ] |
| dlp.srv.world +----------+----------+ node01.srv.world |
| | | |
+----------------------+ +----------------------+

Add oVirt repository on the oVirt Node and Install required packages.

[root@node01 ~]# yum -y install http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm
[root@node01 ~]# yum -y install qemu-kvm libvirt virt-install bridge-utils vdsm
[root@node01 ~]# systemctl start libvirtd
[root@node01 ~]# systemctl enable libvirtd

Access to [https://(oVirt admin server’s hostname or IP address)/] with Web browser on Client Host and then click [Administration Portal].

Distributed Configuration

Tue, 25 May 2021 00:00:00 +0000

Configure Storage Clustering with GlusterFS.

For example, create a distributed volume with 2 Nodes.

This example shows to use 2 Nodes but it’s possible to use more than 3 Nodes.

|
+----------------------+ | +----------------------+
| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |
| node01.srv.world +----------+----------+ node02.srv.world |
| | | |
+----------------------+ +----------------------+
 ⇑ ⇑
 file1, file3 ... file2, file4 ...

It is strongly recommended to use partitions for GlusterFS volumes that are different from the / partition. On this example, it shows settings on the environment that all nodes has [sdb1] and mount it to [/glusterfs].

Add Data Domains

Fri, 12 Oct 2018 00:00:00 +0000

Add Data Domains which manage Virtual machine Disks. It’s possible for Data Domain Storage to use NFS or iSCSI, GlusterFS and so on. This example is based on the environment like follows.

+----------------------+ | +----------------------+
| [ oVirt Management ] |10.0.0.30 | 10.0.0.51| [ oVirt Node ] |
| dlp.srv.world +----------+----------+ node01.srv.world |
| | | | |
+----------------------+ | +----------------------+
 |
+----------------------+ |
| [ NFS server ] |10.0.0.35 |
| nfs.srv.world +----------+
| |
+----------------------+

Install NFS Server, refer to here .

Configure share for Data Domain on NFS Server. (use [/var/lib/ovirt/data] for share on this example)

GlusterFS Client

Tue, 25 May 2021 00:00:00 +0000

Configure GlusterFS Client to mount GlusterFS volumes.

To mount with GlusterFS Native, Set like follows.

The example below shows to mount [vol_distributed] volume to [/mnt].

[root@client ~]# yum -y install centos-release-gluster9
[root@client ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-Gluster-9.repo
[root@client ~]# yum --enablerepo=centos-gluster9 -y install glusterfs glusterfs-fuse
# OK to specify any target nodes in cluster
[root@client ~]# mount -t glusterfs node01.srv.world:/vol_distributed /mnt
[root@client ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs tmpfs 1.9G 8.6M 1.9G 1% /run
tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 26G 1.7G 25G 7% /
/dev/vda1 xfs 1014M 319M 696M 32% /boot
tmpfs tmpfs 379M 0 379M 0% /run/user/0
node01.srv.world:/vol_distributed fuse.glusterfs 52G 3.8G 49G 8% /mnt
# verify read and write
[root@client ~]# echo "Gluster write test" > /mnt/testfile.txt
[root@client ~]# cat /mnt/testfile.txt
Gluster write test

摘自 GlusterFS 9 : GlusterFS Client · Server World，版权归原作者所有，此处仅作个人学习整理。

Add ISO Domains

Fri, 12 Oct 2018 00:00:00 +0000

Add ISO Domains which manage Virtual machine Images. It’s possible for ISO Domain Storage to use NFS or iSCSI, GlusterFS and so on. This example is based on the environment like follows.

+----------------------+ | +----------------------+
| [ oVirt Management ] |10.0.0.30 | 10.0.0.51| [ oVirt Node ] |
| dlp.srv.world +----------+----------+ node01.srv.world |
| | | | |
+----------------------+ | +----------------------+
 |
+----------------------+ |
| [ NFS server ] |10.0.0.35 |
| nfs.srv.world +----------+
| |
+----------------------+

Install NFS Server, refer to here .

Configure share for ISO Domain on NFS Server. (use [/var/lib/ovirt/iso] for share on this example)

GlusterFS + NFS-Ganesha

Tue, 25 May 2021 00:00:00 +0000

Install NFS-Ganesha and integrate with GlusterFS to mount Gluster Volume with NFS protocol. The supported NFS protocols by NFS-Ganesha are v3, v4.0, v4.1, pNFS.

For example, Configure NFS Export setting to a Gluster Volume [vol_distributed] like an example of the link here .

Disable NFS feature in Gluster first.

The NFS feature in Gluster is officially deprecated.

Also if NFS server is running, stop and disable it, too.

# OK if [nfs.disable: on] (default setting)
[root@node01 ~]# gluster volume get vol_distributed nfs.disable
Option Value
------ -----
nfs.disable on
# if [nfs.disable: off], turn to disable
[root@node01 ~]# gluster volume set vol_distributed nfs.disable on
volume set: success
# if NFS server is running, disable it
[root@node01 ~]# systemctl disable --now nfs-server

Install and Configure NFS-Ganesha on a Node in GlusterFS Cluster.

Create Virtual Machines

Fri, 12 Oct 2018 00:00:00 +0000

Create Virtual Machine on oVirt Web Admin Portal. This example is based on the environment below.

+----------------------+ | +----------------------+
| [ oVirt Management ] |10.0.0.30 | 10.0.0.51| [ oVirt Node ] |
| dlp.srv.world +----------+----------+ node01.srv.world |
| | | | |
+----------------------+ | +----------------------+
 |
+----------------------+ |
| [ NFS server ] |10.0.0.35 |
| nfs.srv.world +----------+
| |
+----------------------+

Download ISO file you’d like to install for guestOS on oVirt admin Server. Next, upload it like follows.

# command : engine-iso-uploader -i [ISO Domain Name] upload [ISO File]
[root@dlp ~]# engine-iso-uploader -i ISO_DOMAIN upload CentOS-7-x86_64-DVD-1804.iso
Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort):
# oVirt admin password
Uploading, please wait...
INFO: Start uploading CentOS-7-x86_64-DVD-1804.iso
Uploading: [########################################] 100%
INFO: CentOS-7-x86_64-DVD-1804.iso uploaded successfully

Access to oVirt Admin Portal and Click [Compute] - [Virtual Machines] on the left pane.

GlusterFS + SMB

Tue, 25 May 2021 00:00:00 +0000

Configure GlusterFS volume to enable SMB protocol.

For example, Configure SMB setting to a Gluster Volume [vol_distributed] like an example of the link here .

Configure GlusterFS to enable SMB setting on a Node in GlusterFS Cluster.

[root@node01 ~]# yum -y install centos-release-samba411
[root@node01 ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-Samba-411.repo
[root@node01 ~]# yum --enablerepo=centos-samba411 -y install samba ctdb samba-vfs-glusterfs
# stop the target Gluster volume and change settings
[root@node01 ~]# gluster volume stop vol_distributed
Stopping volume will make its data inaccessible. Do you want to continue? (y/n)
y
volume stop: vol_distributed: success
[root@node01 ~]# gluster volume set vol_distributed user.smb enable
volume set: success
[root@node01 ~]# gluster volume set vol_distributed performance.write-behind off
volume set: success
[root@node01 ~]# gluster volume set vol_distributed group samba
volume set: success
[root@node01 ~]# vi /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
# line 25 : change to the target Gluster volume name
META="
vol_distributed
"
[root@node01 ~]# vi /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh
# line 13 : change to the target Gluster volume name
META="
vol_distributed
"
# start Gluster volume
[root@node01 ~]# gluster volume start vol_distributed
volume start: vol_distributed: success
# with the settings above, following mounting is done automatically
[root@node01 ~]# df -h /gluster/lock
Filesystem Size Used Avail Use% Mounted on
node01.srv.world:/vol_distributed.tcp 52G 3.9G 49G 8% /gluster/lock
[root@node01 ~]# tail -1 /etc/fstab
node01.srv.world:/vol_distributed /gluster/lock glusterfs _netdev,transport=tcp,xlator-option=*client*.ping-timeout=10 0 0
[root@node01 ~]# vi /etc/ctdb/nodes
# create new # write all Nodes that configure target Gluster volume 10.0.0.51 10.0.0.52
[root@node01 ~]# vi /etc/ctdb/public_addresses
# create new # set virtual IP address for SMB access # [eth0] means network interface name ⇒ replace to your environment 10.0.0.59/24 eth0
[root@node01 ~]# systemctl enable --now ctdb
# confirm status
[root@node01 ~]# ctdb status
Number of nodes:2
pnn:0 10.0.0.51 OK (THIS NODE)
pnn:1 10.0.0.52 DISCONNECTED|UNHEALTHY|INACTIVE
Generation:843141804
Size:1
hash:0 lmaster:0
Recovery mode:NORMAL (0)
Recovery master:0
[root@node01 ~]# ctdb ip
Public IPs on node 0
10.0.0.59 0

Configure Samba.

Install Xen

Mon, 25 Jan 2016 00:00:00 +0000

Install Xen to build Virtualization Environment.

Enable CentOS Xen Repository and install some packgaes.

[root@dlp ~]# yum -y install centos-release-xen
[root@dlp ~]# sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-Xen.repo
[root@dlp ~]# yum --enablerepo=centos-virt-xen -y update kernel
[root@dlp ~]# yum --enablerepo=centos-virt-xen -y install xen
[root@dlp ~]# vi /etc/default/grub
# change memory amount for Domain0 (specify proper value on your system)
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=
4096M
,max:
4096M
cpuinfo com1=115200,8n1 .....
[root@dlp ~]# /bin/grub-bootxen.sh
[root@dlp ~]# reboot
# display info
[root@dlp ~]# xl info
host : dlp.srv.world
release : 3.18.21-17.el7.x86_64
version : #1 SMP Fri Dec 18 18:04:14 UTC 2015
machine : x86_64
nr_cpus : 6
max_cpu_id : 5
nr_nodes : 1
cores_per_socket : 1
threads_per_core : 1
cpu_mhz : 2594
hw_caps : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
virt_caps : hvm
total_memory : 12287
free_memory : 8054
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 6
xen_extra : .0-9.el7
xen_version : 4.6.0-9.el7
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Thu Jan 14 15:35:35 2016 +0000 git:6e8597a-dirty
xen_commandline : placeholder dom0_mem=4096M,max:4096M cpuinfo com1=115200,8n1 
 console=com1,tty loglvl=all guest_loglvl=all
cc_compiler : gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
cc_compile_by : mockbuild
cc_compile_domain : centos.org
cc_compile_date : Wed Jan 20 12:25:53 UTC 2016
xend_config_format : 4

Configure Bridge networking for KVM virtual machine.

GlusterFS Geo Replication

Tue, 25 May 2021 00:00:00 +0000

Configure GlusterFS Geo Replication to create a replication volume of a GlusterFS volume on a remote site. This configuration is general Primary-Replica (read only replica) settings.

For example, Configure Geo Replication to a Gluster Volume [vol_distributed] like an example of the link here .

|
+----------------------+ | +----------------------+
| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.53| [GlusterFS Server#3] |
| node01.srv.world +----------+----------+ node03.srv.world |
| | | | |
+----------------------+ | +----------------------+
 [vol_distributed] | [secondary-vol_distributed]
+----------------------+ | +----------------------+
| [GlusterFS Server#2] |10.0.0.52 | 10.0.0.54| [GlusterFS Server#4] |
| node02.srv.world +----------+----------+ node04.srv.world |
| | | |
+----------------------+ +----------------------+

Create a volume for replication on Replica Nodes (node03, node04), refer to here . It proceeds with a volume named [secondary-vol_distributed] on this example.

Create a Virtual Machine

Mon, 25 Jan 2016 00:00:00 +0000

Install GuestOS and create a Virtual Machine. This example shows to install CentOS 7.

This example shows to use Libvirt, so install it first.

# enable CentOS Xen
[root@dlp ~]# yum --enablerepo=centos-virt-xen -y install libvirt libvirt-daemon-xen virt-install
[root@dlp ~]# systemctl start libvirtd
[root@dlp ~]# systemctl enable libvirtd

Create a Virtual Machine with Para-Virtualization.

# create a directory for images
[root@dlp ~]# mkdir -p /var/xen/images
[root@dlp ~]# virt-install \ --connect xen:/// \ --paravirt \ --name centos7 \ --ram 4096 \ --disk path=/var/xen/images/centos7.img,size=20 \ --vcpus 2 \ --os-type linux \ --os-variant rhel7 \ --network bridge=br0 \ --graphics none \ --location 'http://ftp.iij.ad.jp/pub/linux/centos/7/os/x86_64/' \ --extra-args 'text console=com1 utf8 console=hvc0'
Starting install...
# installation starts

After finishing installation, login prompt of GuestOS is show like follows.

Set Quota

Tue, 25 May 2021 00:00:00 +0000

It’s possible to limit Gluster volume usage or directory usage on a Gluster volume to set quota.

For example, Set quota to a Gluster Volume [vol_distributed] like an example of the link here .

Enable quota on taget GlusterFS volume you’d like to set.

[root@node01 ~]# gluster volume quota vol_distributed enable
volume quota : success
[root@node01 ~]# gluster volume info
Volume Name: vol_distributed
Type: Distribute
Volume ID: 31b69330-bf38-4540-8524-2c99184993a0
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: node01:/glusterfs/distributed
Brick2: node02:/glusterfs/distributed
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
cluster.enable-shared-storage: enable
# to disable, set like follows
[root@node01 ~]# gluster volume quota vol_distributed disable

Set quota to the entire volume.

Install VirtualBox

Sat, 23 Jan 2016 00:00:00 +0000

Install Oracle VM VirtualBox to configure Virtualization Environment.

Install some packages and reboot.

# install from EPEL
[root@dlp ~]# yum --enablerepo=epel -y install gcc make glibc kernel-headers kernel-devel dkms
[root@dlp ~]# yum -y update kernel
[root@dlp ~]# reboot

Install VirtualBox.

[root@dlp ~]# curl http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo -o /etc/yum.repos.d/virtualbox.repo
[root@dlp ~]# yum search virtualbox
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: www.ftp.ne.jp
 * extras: www.ftp.ne.jp
 * updates: www.ftp.ne.jp
=========================== N/S matched: virtualbox ============================
VirtualBox-4.3.x86_64 : Oracle VM VirtualBox
VirtualBox-5.0.x86_64 : Oracle VM VirtualBox

 Name and summary matches only, use "search all" for everything.
[root@dlp ~]# yum -y install VirtualBox-5.0
[root@dlp ~]# VBoxManage -v
5.0.14r105127

Install extension pack to use VRDP (Virtual Remote Desktop Protocol).

Add Nodes (Bricks)

Tue, 25 May 2021 00:00:00 +0000

Add Nodes (Bricks) to existing Cluster.

For example, Add a Node [node03] to the existing Cluster like follows .

|
+----------------------+ | +----------------------+
| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |
| node01.srv.world +----------+----------+ node02.srv.world |
| | | | |
+----------------------+ | +----------------------+
 ⇑ | ⇑
 file1, file3 ... | file2, file4 ...
 |
+----------------------+ |
| [GlusterFS Server#3] |10.0.0.53 |
| node03.srv.world +----------+
| |
+----------------------+

Install GlusterFS to a New Node, refer to here , and then Create a directory for GlusterFS volume on the same Path with other Nodes.

Create a Virtual Machine

Sat, 23 Jan 2016 00:00:00 +0000

Install GuestOS and create a Virtual Machine. This example shows to install CentOS 7.

Create a Virtual Machine.

# create a directory for VM
[root@dlp ~]# mkdir /var/vbox
# create a VM
[root@dlp ~]# VBoxManage createvm \ --name CentOS_7 \ --ostype RedHat_64 \ --register \ --basefolder /var/vbox
Virtual machine 'CentOS_7' is created and registered.
UUID: bdceab21-504f-47c2-9107-17ec778dbcf3
Settings file: '/var/vbox/CentOS_7/CentOS_7.vbox'
# modify VM
[root@dlp ~]# VBoxManage modifyvm CentOS_7 \ --cpus 4 \ --memory 4096 \ --nic1 bridged \ --bridgeadapter1 eth0 \ --boot1 dvd \ --vrde on \ --vrdeport 5001
# configure storage for VM
[root@dlp ~]# VBoxManage storagectl CentOS_7 --name "CentOS_7_SATA" --add sata
[root@dlp ~]# VBoxManage createhd \ --filename /var/vbox/CentOS_7/CentOS_7.vdi \ --size 20480 \ --format VDI \ --variant Standard
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: dc515d33-fa38-4c70-92c7-6f2f17a39487
[root@dlp ~]# VBoxManage storageattach CentOS_7 \ --storagectl CentOS_7_SATA \ --port 1 \ --type hdd \ --medium /var/vbox/CentOS_7/CentOS_7.vdi
# configure DVD drive for VM (the example specifies an ISO file)
[root@dlp ~]# VBoxManage storageattach CentOS_7 \ --storagectl CentOS_7_SATA \ --port 0 \ --type dvddrive \ --medium /tmp/CentOS-7-x86_64-DVD-1511.iso
# confirm settings for VM
[root@dlp ~]# VBoxManage showvminfo CentOS_7
Name: CentOS_7
Groups: /
Guest OS: Red Hat (64-bit)
UUID: bdceab21-504f-47c2-9107-17ec778dbcf3
Config file: /var/vbox/CentOS_7/CentOS_7.vbox
Snapshot folder: /var/vbox/CentOS_7/Snapshots
Log folder: /var/vbox/CentOS_7/Logs
Hardware UUID: bdceab21-504f-47c2-9107-17ec778dbcf3
Memory size: 4096MB
Page Fusion: off
VRAM size: 8MB
CPU exec cap: 100%
HPET: off
Chipset: piix3
Firmware: BIOS
Number of CPUs: 4
PAE: on
Long Mode: on
CPUID Portability Level: 0
.....
.....

Start Virtual Machine.

Remove Nodes (Bricks)

Tue, 25 May 2021 00:00:00 +0000

Remove Nodes (Bricks) from existing Cluster.

For example, Remove a Node [node03] from the existing Cluster like follows .

|
+----------------------+ | +----------------------+
| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |
| node01.srv.world +----------+----------+ node02.srv.world |
| | | | |
+----------------------+ | +----------------------+
 ⇑ | ⇑
 file1, file3 ... | file2, file4 ...
 |
+----------------------+ |
| [GlusterFS Server#3] |10.0.0.53 |
| node03.srv.world +----------+
| |
+----------------------+

Remove a New Node from existing Cluster on a node. (OK on any existing node except removing target)

# confirm volume info
[root@node01 ~]# gluster volume info
Volume Name: vol_distributed
Type: Distribute
Volume ID: 9aad84cc-4670-4655-82c8-1071a3fd1b9d
Status: Started
Snapshot Count: 0
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: node01:/glusterfs/distributed
Brick2: node02:/glusterfs/distributed
Brick3: node03:/glusterfs/distributed
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
cluster.enable-shared-storage: enable
# start removing node from volume
# rebalance volume is also run
[root@node01 ~]# gluster volume remove-brick vol_distributed node03:/glusterfs/distributed start
It is recommended that remove-brick be run with cluster.force-migration option disabled to prevent possible data corruption. Doing so will ensure that files that receive writes during migration will not be migrated and will need to be manually copied after the remove-brick commit operation. Please check the value of the option and update accordingly.
Do you want to continue with your current cluster.force-migration settings? (y/n) y
volume remove-brick start: success
ID: 91c7e53e-c6d3-4599-bf30-bd8f187b86b8
# confirm status
[root@node01 ~]# gluster volume remove-brick vol_distributed node03:/glusterfs/distributed status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
 node03 0 0Bytes 0 0 0 completed 0:00:00
# after [status] turning to [completed], commit removing
[root@node01 ~]# gluster volume remove-brick vol_distributed node03:/glusterfs/distributed commit
volume remove-brick commit: success
Check the removed bricks to ensure all files are migrated.
If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick.
# confirm volume info
[root@node01 ~]# gluster volume info
Volume Name: vol_distributed
Type: Distribute
Volume ID: 9aad84cc-4670-4655-82c8-1071a3fd1b9d
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: node01:/glusterfs/distributed
Brick2: node02:/glusterfs/distributed
Options Reconfigured:
performance.client-io-threads: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
cluster.enable-shared-storage: enable

摘自 GlusterFS 9 : Remove Nodes (Bricks) · Server World，版权归原作者所有，此处仅作个人学习整理。

Install GuestAdditions

Sat, 23 Jan 2016 00:00:00 +0000

Install GuestAdditions into a Virtual Machine to improve Virtual Machine’s System Performance. The example below is to install GuestAdditions on a Virtual Machine “CentOS_7”.

Attach GuestAdditions’ disk to the VM on VirtualBox Host.

[root@dlp ~]# VBoxManage storageattach CentOS_7 \ --storagectl CentOS_7_SATA \ --port 0 \ --type dvddrive \ --medium /usr/share/virtualbox/VBoxGuestAdditions.iso

Install GuestAdditions on the VM.

# install some packgaes
[root@guest ~]# yum -y update kernel
[root@guest ~]# reboot
[root@guest ~]# yum -y install gcc make bzip2 kernel-devel
[root@guest ~]# mount /dev/cdrom /mnt
[root@guest ~]# cd /mnt
[root@guest mnt]# ./VBoxLinuxAdditions.run
[root@guest mnt]# reboot

After installing GuestAdditions, System Performance is improved.

Replication Configuration

Tue, 25 May 2021 00:00:00 +0000

Configure Storage Clustering with GlusterFS.

For example, Create a Replication volume with 3 Nodes.

It’s possible to create Replication volume with 2 Nodes but it’s not recommended because split-brain syndrome maybe happens on [replica 2 volume]. As a countermeasure for split-brain, it is recommended to create volume with more than 3 Nodes or set [arbiter volume].

|
+----------------------+ | +----------------------+
| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |
| node01.srv.world +----------+----------+ node02.srv.world |
| | | | |
+----------------------+ | +----------------------+
 |
+----------------------+ |
| [GlusterFS Server#3] |10.0.0.53 |
| node03.srv.world +----------+
| |
+----------------------+

Create a Virtual Machine (GUI)

Sat, 23 Jan 2016 00:00:00 +0000

If you installed Desktop Environment , it’s possible to create Virtual Machines on GUI. This example shows to install Windows Server 2012 R2 on GUI.

Start [Applications] - [System Tools] - [Oracle VM VirtulBox].

Click “New” button which is on upper-left.

Select the OS type for Virtual Machine.

Specify amount of memory for Virtual Machine.

It’s the settings for disks of Virtual Machine. This example shows to select “Create a virtual hard disk now”.

Distributed + Replication

Tue, 25 May 2021 00:00:00 +0000

Configure Storage Clustering with GlusterFS.

For example, Create a Distributed + Replication volume with 6 Nodes. Also Configure Arbiter Volume in order to avoid Split Brain.

|
+----------------------+ | +----------------------+
| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.54| [GlusterFS Server#4] |
| node01.srv.world +----------+----------+ node04.srv.world |
| | | | |
+----------------------+ | +----------------------+
 |
+----------------------+ | +----------------------+
| [GlusterFS Server#2] |10.0.0.52 | 10.0.0.55| [GlusterFS Server#5] |
| node02.srv.world +----------+----------+ node05.srv.world |
| | | | |
+----------------------+ | +----------------------+
 |
+----------------------+ | +----------------------+
| [GlusterFS Server#3] |10.0.0.53 | 10.0.0.56| [GlusterFS Server#6] |
| node03.srv.world +----------+----------+ node06.srv.world |
| | | |
+----------------------+ +----------------------+
 ⇑ ⇑
 file1, file3 ... file2, file4 ...

Install VMware Player

Sat, 10 Jan 2015 00:00:00 +0000

Install VMware Workstation Player. Desktop Environment is required to use it .

Install some packages first.

[root@dlp ~]# yum -y install gcc kernel-headers kernel-devel

Download the latest VMware Workstation Player first from VMware’ site bellow and upload it on your server. Next, install it like follows.

⇒ https://my.vmware.com/web/vmware/downloads

[root@dlp ~]# sh ./VMware-Player-12.1.1-3770994.x86_64.bundle
Extracting VMware Installer...done.
You must accept the VMware Player End User License Agreement to
continue. Press Enter to proceed.
# Enter
...
...
...
Do you agree? [yes/no]:
yes
# agree to the License
You must accept the VMware OVF Tool component for Linux End User
License Agreement to continue. Press Enter to proceed.
# Enter
...
...
...
Do you agree? [yes/no]:
yes
# agree to the License
Would you like to check for product updates on startup? [yes]:
# Enter
Would you like to help make VMware software better by sending
anonymous system data and usage statistics to VMware? [yes]:
# select Yes or No
Enter license key. (optional) You can enter this information later.:
# Enter
The product is ready to be installed. Press Enter to begin
installation or Ctrl-C to cancel.
# Enter
Installing VMware Player 12.1.1
 Configuring...
[#####################################################] 100%
Installation was successful.

摘自 Install VMware Player · Server World，版权归原作者所有，此处仅作个人学习整理。

Create a Virtual Machine

Sat, 10 Jan 2015 00:00:00 +0000

Create Virtula Machine on VMware Player. This example shows to install Windows Server 2012 R2. Desktop Environment is required to this operation .

Open [Applications] - [System Tools] - [VMware Player ].

Click ‘Create a New Virtual machine’.

Select installation source of GuestOS.

Select a kind of guest OS.

Specify the name and location for Virtual Machine.

Nested Virtualization

Sat, 10 Jan 2015 00:00:00 +0000

Configure Nested Virtualization function and Install HyperVisor like KVM on GuestOS on VMware Player. It’s possible to nest virtualization to test various apprications. Your CPU is required to have Intel VT-x/EPT or AMD-V/RVI. ⇒ http://ark.intel.com/Products/VirtualizationTechnology/

Open the “Virtual Machine Settings” of a Virtual Machine you’d like to configure Nested Virtualization. After opening it, move to “Processors” section, then configure the settings like follows. It’s OK for all.

摘自 Nested Virtualization · Server World，版权归原作者所有，此处仅作个人学习整理。

Markdown 写作速查

Mon, 05 Oct 2026 00:00:00 +0000

写文章只需要记住很少几个符号。下面这段把常用的语法和它的渲染结果放在一起，忘了的时候回来翻一眼。

行内格式

写法	效果
`加粗`	加粗
`斜体`	斜体
`代码`	`代码`
`~~删除线~~`	~~删除线~~
`[链接](https://example.com)`	链接

标题

用 # 到 #### 表示四级标题。正文里建议只用 ## 和 ###，因为 ## 会自动出现在右侧目录中，一级标题留给文章名。

关于

Mon, 05 Oct 2026 00:00:00 +0000

这里是一个写字的地方。

内容以文字为主：读书笔记、技术备忘、偶尔的胡思乱想。没有评论区，也没有访问统计——如果你想聊点什么，发邮件就好。

关于 CentOS 7 笔记

笔记栏目的内容整理自 Server World 的 CentOS 7 配置示例，只保留文字部分，重排为便于阅读的格式。原文版权归 Server World 所有，每篇末尾都保留了出处链接，此处仅作个人离线查阅与学习整理，不作任何商业用途。

这里开始写字

Mon, 05 Oct 2026 00:00:00 +0000

这台机器是搬瓦工的入门款：两核 CPU、1 GB 内存、20 GB 磁盘。跑数据库太挤，跑容器又有点浪费，最后选了最朴素的方案——静态站点。

为什么是静态的

静态站的意思是：文章写在 Markdown 文件里，构建时一次性生成 HTML，服务器上只剩下一堆纯文本文件。没有数据库，没有后台进程，没有 PHP 或 Node 在后台常驻。