Monthly Archives: October 2009

GPFS commands to create NSDs, filesystems, add and delete

Posted on by
Reply

Put this line in a file ‘nsd.txt’: hdiskpower5::::

[root@nad0019aixd06/usr/lpp/mmfs/bin]# mmcrnsd -F nsd.txt
mmcrnsd: Processing disk hdiskpower5
mmcrnsd: Propagating the changes to all affected nodes.
This is an asynchronous process.

_________________________________________________________

[root@nad0019aixd06/usr/lpp/mmfs/bin]# mmlsnsd

File system Disk name Primary node Backup node
—————————————————————————
backups satadrive (directly attached)
oragpfs gpfs1nsd (directly attached)
oragpfs gpfs2nsd (directly attached)
oragpfs gpfs9nsd (directly attached)
testgpfs nsd_test_5 (directly attached)
testgpfs nsd_test_6 (directly attached)
(free disk) gpfs11nsd (directly attached)
—————————————————————————————————————————

mmcrfs /oragpfs -F nsd.txt (only use this to add a new filesystem)

mmadddisk /dev/oragpfs -F nsd.txt

# mmcrnsd -F nsd.txt
mmcrnsd: Processing disk hdiskpower5
mmcrnsd: Processing disk hdiskpower9
mmcrnsd: Processing disk hdiskpower10
mmcrnsd: Propagating the changes to all affected nodes.
This is an asynchronous process.
# mmadddisk /dev/oragpfs -F nsd.txt

The following disks of oragpfs will be formatted on node nad0019aixd07:
gpfs12nsd: size 104857600 KB
gpfs13nsd: size 104857600 KB
gpfs14nsd: size 104857600 KB
Extending Allocation Map
Completed adding disks to file system oragpfs.
mmadddisk: Propagating the changes to all affected nodes.
This is an asynchronous process.
__________________________________________________________________________________

[root@nad0019aixd06/usr/lpp/mmfs/bin]# mmlsfs oragpfs
flag value description
—- ————– ——————————————————
-s roundRobin Stripe method
-f 16384 Minimum fragment size in bytes
-i 512 Inode size in bytes
-I 16384 Indirect block size in bytes
-m 1 Default number of metadata replicas
-M 1 Maximum number of metadata replicas
-r 1 Default number of data replicas
-R 1 Maximum number of data replicas
-j cluster Block allocation type
-D posix File locking semantics in effect
-k posix ACL semantics in effect
-a -1 Estimated average file size
-n 8 Estimated number of nodes that will mount file system
-B 524288 Block size
-Q none Quotas enforced
none Default quotas enabled
-F 2000000 Maximum number of inodes
-V 8.01 File system version. Highest supported version: 8.02
-u yes Support for large LUNs?
-z no Is DMAPI enabled?
-E yes Exact mtime mount option
-S no Suppress atime mount option
-d gpfs1nsd;gpfs2nsd;gpfs9nsd Disks in file system
-A yes Automatic mount option
-o none Additional mount options
-T /oragpfs Default mount point

# mmdf oragpfs -d
disk disk size failure holds holds free KB free KB
name in KB group metadata data in full blocks in fragments
————— ————- ——– ——– —– ——————- ——————-
gpfs14nsd 104857600 -1 yes yes 104838144 (100%) 496 ( 0%)
gpfs13nsd 104857600 -1 yes yes 104838144 (100%) 496 ( 0%)
gpfs9nsd 47185920 -1 yes yes 9341952 (20%) 72944 ( 0%)
gpfs12nsd 104857600 -1 yes yes 104838144 (100%) 496 ( 0%)
gpfs2nsd 104857600 1 yes yes 22041600 (21%) 127328 ( 0%)
gpfs1nsd 104857600 1 yes yes 22012416 (21%) 130784 ( 0%)
————- ——————- ——————-
(total) 571473920 367910400 (64%) 332544 ( 0%)

You must change maximum number of inodes to use space. The formula in the manual lists

maximum number of files = (total file system space/2) / (inode size + subblock size)

8,929,280 = (571473920/2) / (524288/16384)

# mmchfs oragpfs -F 8929280

# mmlsfs oragpfs
flag value description
—- ————– —————————————————–
-s roundRobin Stripe method
-f 16384 Minimum fragment size in bytes
-i 512 Inode size in bytes
-I 16384 Indirect block size in bytes
-m 1 Default number of metadata replicas
-M 1 Maximum number of metadata replicas
-r 1 Default number of data replicas
-R 1 Maximum number of data replicas
-j cluster Block allocation type
-D posix File locking semantics in effect
-k posix ACL semantics in effect
-a -1 Estimated average file size
-n 8 Estimated number of nodes that will mount file system
-B 524288 Block size
-Q none Quotas enforced
none Default quotas enabled
-F 8929280 Maximum number of inodes
-V 8.01 File system version. Highest supported version: 8.02
-u yes Support for large LUNs?
-z no Is DMAPI enabled?
-E yes Exact mtime mount option
-S no Suppress atime mount option
-d gpfs1nsd;gpfs2nsd;gpfs9nsd;gpfs12nsd;gpfs13nsd;gpfs14nsd Disks in file system
-A yes Automatic mount option
-o none Additional mount options
-T /oragpfs Default mount point
#

No change in FS will be seen but it will allow GPFS to grow file size as needed.

______________________________________________________________________________________

To add a disk that is already defined as an nsd:

# mmadddisk orabackup nsd_0314_04::::

The following disks of orabackup will be formatted on node nad0019aixp02s1:
nsd_0314_04: size 262144000 KB
Extending Allocation Map
8 % complete on Fri Nov 16 13:23:21 2007
19 % complete on Fri Nov 16 13:23:26 2007
31 % complete on Fri Nov 16 13:23:31 2007
42 % complete on Fri Nov 16 13:23:36 2007
53 % complete on Fri Nov 16 13:23:41 2007
62 % complete on Fri Nov 16 13:23:46 2007
70 % complete on Fri Nov 16 13:23:51 2007
82 % complete on Fri Nov 16 13:23:56 2007
97 % complete on Fri Nov 16 13:24:01 2007
100 % complete on Fri Nov 16 13:24:02 2007
Completed adding disks to file system orabackup.
mmadddisk: Propagating the changes to all affected nodes.
This is an asynchronous process.

When you add a new disk to an existing filesystem, it takes a little while for the free space to show up.

How to remove a disk from gpfs:

# ./mmlsnsd

File system Disk name Primary node Backup node
—————————————————————————
gpfs10nsd gpfs10nsd (directly attached)
gpfs11nsd gpfs11nsd (directly attached)
gpfs7nsd gpfs7nsd (directly attached)
gpfs9nsd gpfs9nsd (directly attached)
orabackup gpfs1nsd (directly attached)
orabackup gpfs2nsd (directly attached)
orabackup gpfs3nsd (directly attached)
orabackup gpfs4nsd (directly attached)
orabackup gpfs5nsd (directly attached)
oracrsocr gpfs8nsd (directly attached)
(free disk) gpfs12nsd (directly attached)
(free disk) gpfs6nsd (directly attached)

# ./mmdeldisk orabackup “gpfs1nsd;gpfs2nsd;gpfs4nsd” -r
Deleting disks …
Scanning system storage pool
GPFS: 6027-589 Scanning file system metadata, phase 1 …
70 % complete on Mon Oct 13 16:26:00 2008
100 % complete on Mon Oct 13 16:26:01 2008
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 2 …
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 3 …
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 4 …
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-565 Scanning user file metadata …
GPFS: 6027-552 Scan completed successfully.
Checking Allocation Map for storage pool ‘system’
GPFS: 6027-370 tsdeldisk64 completed.
mmdeldisk: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
Restriping orabackup …
GPFS: 6027-589 Scanning file system metadata, phase 1 …
71 % complete on Mon Oct 13 16:26:16 2008
100 % complete on Mon Oct 13 16:26:17 2008
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 2 …
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 3 …
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-589 Scanning file system metadata, phase 4 …
GPFS: 6027-552 Scan completed successfully.
GPFS: 6027-565 Scanning user file metadata …
GPFS: 6027-552 Scan completed successfully.
Done
#

Create a new GPFS cluster

Posted on by
Reply

1. Create a nodelist file.
# more /home/root/nodelist
10.32.18.98:quorum
10.32.18.99

2. Configure ssh/scp to run as root without a password.

3. Run the mmcrcluster command.

mmcrcluster -n /home/root/nodelist -p d06gpfs -s d07gpfs -r /usr/bin/ssh -R /usr/bin/scp

Tue Jun 24 16:24:03 CDT 2008: 6027-1664 mmcrcluster: Processing node d06gpfs
Tue Jun 24 16:24:05 CDT 2008: 6027-1664 mmcrcluster: Processing node d07gpfs
mmcrcluster: Command successfully completed
mmcrcluster: 6027-1371 Propagating the changes to all affected nodes.
This is an asynchronous process.
# mmlscluster

GPFS cluster information
========================
GPFS cluster name: d06gpfs
GPFS cluster id: 729603352964457971
GPFS UID domain: d06gpfs
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers:
———————————–
Primary server: d06gpfs
Secondary server: d07gpfs

Node number Node name IP address Full node name Remarks
———————————————————————————–
1 d06gpfs 10.32.18.98 d06gpfs quorum node
2 d07gpfs 10.32.18.99 d07gpfs

Server Creation

Posted on by
Reply

1. Have the person needing the server submit a change listing you as the person responsible for server setup. Determine if you have necessary hardware, power, and computer rack location before beginning the work.

2. Submit a change ticket to obtain the IP address with new Server name. Request the cabling and Ethernet hardware necessary to accomplish the task.

3. Create a change ticket requesting the storage necessary from the originating disks, fibers, and cables based on the originating ticket.

4. Place hardware in cabinets, power up and determine if the hardware is seen by hardware management console. Note this may take getting Ethernet cables from the server to the HMC if not already present.

5. Create the LPAR from the Hardware Management Console where the server will be located. Use the spec’s submitted in the originating ticket.

6. Setup the Ethernet adapter (Shared or dedicated). Add adapter thru the HMC from available slot. Add IP address and default route. Check the connection with ping, nfs, or telnet.

7. Load the operating system if you do not already have it loaded. A NIM (Network Install Manager) environment has been setup on NAD0019aixd10 which should have the necessary file sets. Note make certain that the necessary patches are also applied to be certain the environment is up to date with current IBM supported patch sets.

8. Logon the server setting the root password to the standard used by the system administration staff.

9. Load SSL and SSH software assuring your connections is protected.

Change /etc/ssh_config – Setup value # MaxStartup 10 to MaxStartup 500.

10. Setup necessary disk storage space. If EMC be certain the fiber adapters are up to date on microcode levels in order to avoid future problems with access errors.

11. Loaded standard software as required by user requesting the system.

a. SUDOER
b. foglight
c. Setup BUXS directory with all home grown software.

12. Create user base that will use the necessary initially for DBA’s or other personnel to start their software loads.

13. Create user file systems from orginating ticket with required software.

14. Assist with any other software necessary for DBA load.

15. Have users test the application.

16. Reboot server to test everything comes back as required. Add custom scripts to root cron or inittab for rebooting to fix any reboot issues.

17: Secure server:

chuser rlogin=false root

18. In AIX 5.3 for Oracle:

chdev -l aio0 -P -a maxreqs=8192 (requires reboot)
vmo -p -o lru_file_repage=0

vi /etc/rc.tcpip:

Change:
start /usr/sbin/hostmibd “$src_running” “-c buxs”
start /usr/sbin/snmpmibd “$src_running” “-c buxs”
start /usr/sbin/aixmibd “$src_running” “-c buxs”

vi /etc/snmpdv3.conf:

Change all instances of ‘public’ to ‘buxs’

Comment out :
#start /usr/sbin/muxatmd …

vi /etc/inetd.conf:

Comment out:
# ftp ……..
# telnet ……….

vi /etc/inittab:

Comment out fbcheck ( with ‘:’ not, ‘#’ )

18: Set up buxs user with ssh key and initial scripts

19. for oracle:

touch /etc/oratab
chown oracle.dba /etc/oratab

copy from running system (if a client install only): /usr/local/bin files below:

-rwxr-xr-x 207 0 2413 Apr 12 15:17:10 2007 ./dbhome
-rwxr-xr-x 207 0 2545 Apr 12 15:17:14 2007 ./oraenv
-rwxr-xr-x 207 0 2316 Apr 12 15:17:17 2007 ./coraenv

20: Change the smtp server

21. Add to foglight,veritas,nessus

22. Create an entry for error reports to be sent to email.

chdev -l aio0 -P -a autoconfig=’available’

23. Set the maximum # of processes to be really big ( chdev -l sys0 -a maxuproc=90000 )

24. Set /etc/security/limits to be unlimited for oracle

Make sure system will boot

Posted on by
Reply

1. bosboot -ad /dev/hdisk5

root@nad0019aixd09/dev> bootlist -m normal -o
hdisk5 blv=hd5

bootlist -m normal -o

ln rhdisk5 ipldevice ( use ln to create a copy of the raw devices if necessary)

These need to all exist and point to rootvg, rhd5, and rhdiskX ( being whereever the boot image is):

root@nad0019aixd09/dev> ls -l | grep -i ipl
crw-rw—- 1 root system 10, 0 Jan 11 2006 IPL_rootvg
crw-rw—- 2 root system 10, 1 Apr 23 13:02 ipl_blv
crw——- 2 root system 20, 6 Apr 23 12:19 ipldevice

root@nad0019aixd09/dev> ipl_varyon -i

PVNAME BOOT DEVICE PVID VOLUME GROUP ID
hdisk2 NO 00033f6a7c51d6bd0000000000000000 00cdeaea00004c00
hdisk4 NO 00cdeaea38c938d20000000000000000 00cdeaea00004c00
hdisk5 YES 00033f6a7c4c17d40000000000000000 00cdeaea00004c00 <--- this one is important

NIM tricks

Posted on by
Reply

If the NIM connection is acting funny:

server: startsrc -s nimesis
rm /etc/niminfo
(options for adding clients)
lsnim -q define -t standalone
smitty nim_mkmac

smitty nim (configure)
or
niminit -a name=drviop02 -a master=nad0019aixd10

Fastpath to client installer:
smitty nim_client_inst

Set up a VIO server to be more useable

Posted on by
Reply

The special padmin user that IBM tried to make us all use was a big mistake. A vio server is really just an AIX server with a different set of commands. I find it easier to circumvent all of this:

SEED=somegoodhost
scp $SEED:/etc/security/login.cfg /etc/security/login.cfg
chuser su=true login=true rlogin=true shell=/usr/bin/bash root
ln -s /usr/ios/cli/ioscli /usr/sbin/ioscli
chuser shell=/usr/bin/bash groups=staff,buxs rigljoha

Adding Bash to AIX

Posted on by
Reply

In order to have bash as your login on AIX, you have to change /etc/security/login.cfg:


default:
        sak_enabled = false
        logintimes =
        logindisable = 0
        logininterval = 0
        loginreenable = 0
        logindelay = 0




usw:
        shells = /bin/sh,/bin/bsh,/bin/csh,/bin/ksh,/bin/tsh,/bin/ksh93,/usr/bin/sh,/usr/bin/bsh,/usr/bin/csh,/usr/bin/ksh,/usr/bin/tsh,/usr/bin/ksh93,/usr/bin/rksh,/usr/bin/rksh93,/usr/sbin/uucp/uucico,/usr/sbin/sliplogin,/usr/sbin/snappd,/bin/bash
        maxlogins = 32767
        logintimeout = 60
        auth_type = STD_AUTH

Next:

chuser -a shell=/usr/bin/bash rigljoha