My Information
My keybase.io/GPG
Contact Me
@billysmusic
T&Cs
michaelspost.com


Posts and stuff


In order to run a manual command against etcd running inside the K8S cluster you have to pass a few flags
Be very careful with this as you can really mess things up
Get the members of the etcd cluster
kubectl exec -it -n kube-system etcd-k8sn1 -- etcdctl --ca-file=/etc/kubernetes/pki/etcd/ca.crt --cert-file=/etc/kubernetes/pki/etcd/peer.crt --key-file=/etc/kubernetes/pki/etcd/peer.key --endpoints=https://127.0.0.1:2379 member list
Should get you output like:
3c06d56f4e8193e0: name=k8sn3 peerURLs=https://192.168.1.54:2380 clientURLs=https://192.168.1.54:2379 isLeader=false
3cc1193e53313d05: name=k8sn2 peerURLs=https://192.168.1.55:2380 clientURLs=https://192.168.1.55:2379 isLeader=false
a7447908c6035378: name=k8sn1 peerURLs=https://192.168.1.52:2380 clientURLs=https://192.168.1.52:2379 isLeader=false
efd47b5d63640666: name=k8sn4 peerURLs=https://192.168.1.53:2380 clientURLs=https://192.168.1.53:2379 isLeader=true
You can then remove a specific node
kubectl exec -it -n kube-system etcd-k8sn1 -- etcdctl --ca-file=/etc/kubernetes/pki/etcd/ca.crt --cert-file=/etc/kubernetes/pki/etcd/peer.crt --key-file=/etc/kubernetes/pki/etcd/peer.key --endpoints=https://127.0.0.1:2379 member remove a7447908c6035378


In order to specify the UEFI boot parameters in a UCS boot policy you need to be using one of the following disk types In this example we will be booting to a RedHat Virtualization Host (RHVH) in UEFI mode with SAN boot
Configure a boot policy to point to your disk type, in this case SAN then add the UEFI parameters as follows
Examples screenshots are using the 3.2 version of the UCS interface (HTML5), so they may look a bit different if using the old Java client
  1. Select the Servers tab
  2. Expand Servers -> Policies
  3. Expand Boot Policies and select your boot policy
    Example:
  4. Select the LUN/Disk in the Boot Order list
  5. Click Modify/Set Uefi Boot Parameters
  6. For RHVH, set the following (This will be the similar for other Linux distros besides the name)
  7. Boot Loader Name: RHVH
  8. Boot Loader Path: EFI\BOOT\BOOTX64.EFI
    Example:



Do base 10 arithmetic in bash instead of octal 
((newvar = 10#$var + 10))
Remove leading zeros in bash 
newvar=${var#0}
Find processes & threads for users (useful if running into nproc limits) 
ps h -Led -o user | sort | uniq -c | sort -n
Show threads per process for user bob 
ps -o nlwp,pid,lwp,args -u bob | sort -n
Delete files when rm/bash says too many arguments

No xargs
find . -name '*.pdf' -exec rm {} +
Using xargs
find . -name "*.pdf" -print0 | xargs -0 rm
Pretty tree of processes
ps -eo 'pid,args' --forest
Scan/re-scan a fibre channel adapter and scsi host

FC host
echo 1 > /sys/class/fc_host/host/issue_lip
SCSI host
echo "- - -" > /sys/class/scsi_host/host/scan
Individual disk
echo 1 > /sys/class/scsi_disk/0\:0\:0\:0/device/rescan
Check for open tcp ports from bash without telnet or nc

Replace <hostname> and <port> with your host and port you are checking
timeout 1 bash -c "cat < /dev/null > /dev/tcp/<hostname>/<port>"
Check the return result (echo $?) afterward and if it is 0 we had a successful connection, 1 means there was an error connecting before the timeout expired, 124 means the timeout expired and there was no connection

Display info from istheinternetonfire.com in a pretty way

Requires cowsay and the lolcat ruby gem (gem install lolcat)
case $- in *i*)dig +short txt istheinternetonfire.com | cowsay -f moose | lolcat; esac


There doesn't seem to be a way to install LVM directly to a raw device in the CentOS/RedHat installer. With a little bit of custom kickstart we can make it do it. This may work on CentOS 6, but I haven't tested it.

This assumes you have two drives connected to your system, one for boot and one for the system (I'm using VMs, so this is as simple as adding two disks). This also assumes that the disk device names are sd*, so make adjustments according to your system.

My Setup

1st disk - 2GB for boot
2nd disk - 30GB for OS

%pre section of config

In your kickstart config you will need a %pre section that does the following in order:

1. Creates an LVM physical volume on the device. This does a force (-ff) and overwrites any data there so be careful. -y just tells pvcreate to say yes to any "are you sure" messages.

2. Creates an LVM volume group named vg_system on the physical volume

This is all being redirected to /dev/tty1 so you can see the output during %pre

%pre
pvcreate -y -ff /dev/sdb &> /dev/tty1
vgcreate vg_system /dev/sdb &> /dev/tty1
%end

Main section of config

In this main section of your config you will need something similar to this, adjust for the partitions and sizes you want. In order, this will:

1. Wipe out sda and initialize the disk if new

2. Put a single boot partition on it that fills the drive

3. Tells the kickstart to use the existing volume group created in the %pre

4. Creates a logical volume in the volume group for root (/) that fills the disk asside from swap

5. Creates a logical volume in the volume group for swap of 4GB

clearpart --drives=sda --initlabel --all
part /boot --fstype=ext4 --size=1 --grow --ondisk=sda
volgroup vg_system --useexisting
logvol / --fstype=ext4 --name=lv_root --vgname=vg_system --grow --size=1024
logvol swap --name=lv_swap --vgname=vg_system --grow --size=4096 --maxsize=4096

If you don't have a base kickstart config to work with look on a CentOS/RedHat box in root's home directory for an anaconda-ks.cfg which is the kickstart equivalent of how that system was installed



Pre-requisites

Make sure the hostname of the system is set correctly and all firewalls and selinux are disabled (Or configured for the ports specified at the bottom)

Install the packages

pam-devel is necessary if you want to build libnss_winbind.so.2 and use pam/winbind for local auth

yum -y install gcc make wget python-devel gnutls-devel openssl-devel libacl-devel krb5-server krb5-libs krb5-workstation bind bind-libs bind-utils pam-devel ntp openldap-devel ncurses-devel

Download and compile Samba

You can replace -j3 with n+1 CPUs you have available on the system.

wget https://www.samba.org/samba/ftp/stable/samba-4.3.1.tar.gz
tar xvzf samba-4.3.1.tar.gz
cd samba-4.3.1
./configure --enable-selftest
make -j3 && make -j3 install

Run Samba domain creation and configuration

This will create your domain as domain.com using bind as the backend and setting rfc2307 attributes in AD so you can store and retrieve UID/GID in AD.

/usr/local/samba/bin/samba-tool domain provision --realm=domain.com --domain=DOMAIN --server-role=dc --dns-backend=BIND9_DLZ --adminpass "myadpassword" --use-rfc2307

Create an init script for Samba

Edit /etc/init.d/samba4

#! /bin/bash
#
# samba4 start and stop samba service
#
# chkconfig: - 90 10
# description: Activates/Deactivates all samba4 interfaces configured to start at boot time.
#
# config: /usr/local/samba/etc/smb.conf
### BEGIN INIT INFO
# Provides:
# Required-Start: $local_fs $network
# Required-Stop: $local_fs $network
# Should-Start:
# Short-Description: start and stop samba service
# Description: start and stop samba service
### END INIT INFO
# Source function library.
. /etc/init.d/functions

if [ -f /etc/sysconfig/samba4 ]; then
. /etc/sysconfig/samba4
fi

RETVAL=0
CWD=$(pwd)
prog="samba"
prog_path="/usr/local/samba/sbin"
lockfile=/var/lock/subsys/$prog

start() {
        #Samba is already running, exit
        [ -e $lockfile ] && exit 1
        #Start service
        echo -n $"Starting $prog: "
        daemon $prog_path/$prog
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $lockfile
        return $RETVAL
}
stop() {
        # Stop service.
        echo -n $"Shutting down $prog: "
        killproc $prog_path/$prog
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f $lockfile
        return $RETVAL
}

# See how we were called.
case "$1" in
        start)
                start
                RETVAL=$?
                ;;
        stop)
                stop
                RETVAL=$?
                ;;
        status)
                status smbd
                status samba
                RETVAL=$?
                ;;
        restart|reload)
                stop
                start
                ;;
        *)
                echo $"Usage: $0 {start|stop|restart|status}"
                exit 1
esac

exit $RETVAL
chmod 755 /etc/init.d/samba4

Configure Bind/DNS

Configure Bind to allow recursion (external resolving) to local subnets and point to the kerberos key that the samba-tool command generated. Also include the Samba Bind database.

acl "trusted" {
        192.168.1.0/24;
        127.0.0.0/8;
        ::1/128;
};
options {
        listen-on port 53 { any; };
        allow-query { any; };
        tkey-gssapi-keytab "/usr/local/samba/private/dns.keytab";
        allow-recursion {
                /* Only trusted addresses are allowed to use recursion. */
                trusted;
        };
        notify yes;
};
include "/usr/local/samba/private/named.conf";

/usr/local/samba/private/named.conf looks like below. CentOS 6 comes with Bind 9.8 which is the default.

dlz "AD DNS Zone" {
    # For BIND 9.8.0
    database "dlopen /usr/local/samba/lib/bind9/dlz_bind9.so";

    # For BIND 9.9.0
    # database "dlopen /usr/local/samba/lib/bind9/dlz_bind9_9.so";
};

Start named and make sure it is on by default

chkconfig named on
/etc/init.d/named start

Change /etc/resolv.conf to point to itself for DNS

domain domain.com
nameserver 127.0.0.1

Configure kerberos

edit /etc/krb5.conf. Case is important on the REALM here

[libdefaults]
        default_realm = DOMAIN.COM
        dns_lookup_realm = true
        dns_lookup_kdc = true

Configure NTP

Edit /etc/ntp.conf. I commented out the IPv6 stuff as it was causing errors in the logs, not necessary but makes the log nicer. Note the sections on ntpsigndsocket and mssntp, these are required for Windows machines to successfully sync time with this AD server.

restrict default nomodify notrap nopeer noquery
#restrict -6 default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1

ntpsigndsocket /usr/local/samba/var/lib/ntp_signd/

# Hosts on local network are less restricted.
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap kod nopeer mssntp

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

# List of time sources managed by puppet class { "ntp": servers => [ ... ] }
server 0.gentoo.pool.ntp.org version 3 prefer
server 1.gentoo.pool.ntp.org version 3 prefer
server 2.gentoo.pool.ntp.org version 3 prefer
server 3.gentoo.pool.ntp.org version 3 prefer

# Drift file.  Put this in a directory which the daemon can write to.
# No symbolic links allowed, either, since the daemon updates the file
# by creating a temporary in the same directory and then rename()\'ing
# it to the file.
driftfile /var/lib/ntp/drift

# Logging
logfile /var/log/ntp
logconfig =syncevents +clockevents

Start ntpd and make sure it is on by default

chkconfig ntpd on
/etc/init.d/ntpd start

Start it up

Start samba4 and make sure it is on by default

chkconfig samba4 on
/etc/init.d/samba4 start

Firewall

Open the following ports:
UDP 53, 123, 135, 138, 389
TCP 88, 464, 139, 445, 389, 636, 3268, 3269
53 - DNS
88 - Kerberos
123 - NTP
135 - RCP
138 - NetBIOS
139 - NetBIOS session
389 - LDAP
445 - MS directory services
464 - Kerberos passwd
636 - SSL LDAP
3268 - Global catalog
3269 - SSL Global catalog