TF实战丨使用Vagrant安装Tungsten Fabric

Easter79
• 阅读 859

本文为苏宁网络架构师陈刚的原创文章。

01准备测试机

在16G的笔记本没跑起来,就干脆拼凑了一台游戏工作室级别的机器:双路E5-2860v3 CPU,24核48线程,128G DDR4 ECC内存,NVME盘 512G。在上面开5个VM,假装是物理服务器。

· 192.16.35.110 deployer

· 192.16.35.111 tf控制器

· 192.16.35.112 openstack服务器,同时也是计算节点

· 192.16.35.113 k8s master

· 192.16.35.114 k8s的Node k01,同时也是ops的计算节点

直接使用vagrant拉镜像会很慢,就先下载下来:

https://cloud.centos.org/centos/7/vagrant/x86_64/images/

下载对应的VirtualBox.box文件。

然后使用命令, 命名为vagrant的box:

vagrant box add centos/7 CentOS-7-x86_64-Vagrant-2004_01.VirtualBox.box

cat << EEOOFF > vagrantfile
###start 
#-*- mode: ruby -*-
#vi: set ft=ruby :
Vagrant.require_version ">=2.0.3"

#All Vagrant configuration is done below. The "2" in Vagrant.configure
#configures the configuration version (we support older styles for
#backwards compatibility). Please don't change it unless you know what
#you're doing.

ENV["LC_ALL"] = "en_US.UTF-8"

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure("2") do |config|
  #The most common configuration options are documented and commented below.
  #For a complete reference, please see the online documentation at
  #https://docs.vagrantup.com.

  #Every Vagrant development environment requires a box. You can search for
  #boxes at https://atlas.hashicorp.com/search.

  config.vm.box = "geerlingguy/centos7"
  #config.vbguest.auto_update = false
  #config.vbguest.no_remote = true  

  config.vm.define "deployer" do | dp |
    dp.vm.provider "virtualbox" do | v |
      v.memory = "8000"
      v.cpus = 2
    end
    dp.vm.network "private_network", ip: "192.16.35.110", auto_config: true
    dp.vm.hostname = "deployer"
  end

  config.vm.define "tf" do | tf |
    tf.vm.provider "virtualbox" do | v |
      v.memory = "64000"
      v.cpus = 16
    end
    tf.vm.network "private_network", ip: "192.16.35.111", auto_config: true
    tf.vm.hostname = "tf"
  end

  config.vm.define "ops" do | os |
    os.vm.provider "virtualbox" do | v |
      v.memory = "16000"
      v.cpus = 4
    end
    os.vm.network "private_network",ip: "192.16.35.112",  auto_config: true
    os.vm.hostname = "ops"
  end

  config.vm.define "k8s" do | k8 |
    k8.vm.provider "virtualbox" do | v |
      v.memory = "8000"
      v.cpus = 2
    end
    k8.vm.network "private_network", ip: "192.16.35.113", auto_config: true
    k8.vm.hostname = "k8s"
  end

  config.vm.define "k01" do | k1 |
    k1.vm.provider "virtualbox" do | v |
      v.memory = "4000"
      v.cpus = 2
    end
    k1.vm.network "private_network", ip: "192.16.35.114", auto_config: true
    k1.vm.hostname = "k01"
  end

  config.vm.provision "shell", privileged: true, path: "./setup.sh"

end

EEOOFF

cat << EEOOFF > setup.sh
#!/bin/bash
##Setup vagrant vms.
#set -eu

#Copy hosts info
cat <<EOF > /etc/hosts
127.0.0.1 localhost
127.0.1.1 vagrant.vm vagrant

192.16.35.110 deployer
192.16.35.111 tf
192.16.35.112 ops
192.16.35.113 k8s
192.16.35.114 k01

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
EOF

systemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
iptables -P FORWARD ACCEPT

swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab
# swapoff -a && sysctl -w vm.swappiness=0

# setenforce  0 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config  

# modprobe ip_vs_rr
modprobe br_netfilter

yum -y update

# sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
# sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
# yum install -y bridge-utils.x86_64
# modprobe bridge
# modprobe br_netfilter
# Setup system vars

yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools vim chrony python python-setuptools python-pip iproute lrzsz tree git

yum install -y libguestfs-tools libvirt-python virt-install libvirt ansible

pip install wheel --upgrade -i https://mirrors.aliyun.com/pypi/simple/
pip install pip --upgrade -i https://mirrors.aliyun.com/pypi/simple/
pip install ansible  netaddr --upgrade -i https://mirrors.aliyun.com/pypi/simple/

# python-urllib3 should be installed before "pip install requests"
# if install failed, pip uninstall urllib3, then reinstall python-urllib3
# pip uninstall -y urllib3 | true
# yum install -y python-urllib3 
pip install requests -i https://mirrors.aliyun.com/pypi/simple/

systemctl disable libvirtd.service
systemctl disable dnsmasq
systemctl stop libvirtd.service
systemctl stop dnsmasq

if [  -d "/root/.ssh" ]; then
      rm -rf /root/.ssh
fi

ssh-keygen -q -t rsa -N "" -f ~/.ssh/id_rsa

cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod go-rwx ~/.ssh/authorized_keys

# 
# timedatectl set-timezone Asia/Shanghai

if [ -f "/etc/chrony.conf" ]; then
   mv /etc/chrony.conf /etc/chrony.conf.bak
fi

cat <<EOF > /etc/chrony.conf
      allow 192.16.35.0/24
      server ntp1.aliyun.com iburst
      local stratum 10
      logdir /var/log/chrony
      rtcsync
      makestep 1.0 3
      driftfile /var/lib/chrony/drift
EOF

systemctl restart chronyd.service
systemctl enable chronyd.service

echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

if [ ! -d "/var/log/journal" ]; then
  mkdir /var/log/journal
fi

if [ ! -d "/etc/systemd/journald.conf.d" ]; then
  mkdir /etc/systemd/journald.conf.d
fi

cat <<EOF > /etc/systemd/journald.conf.d/99-prophet.conf 
[Journal]
Storage=persistent

Compress=yes

SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000

SystemMaxUse=10G

SystemMaxFileSize=200M

ForwardToSyslog=no

EOF

systemctl restart systemd-journald

EEOOFF

02在所有的节点上安装docker

CentOS

例如:如果pip安装软件的速度很慢,可以考虑使用基于aliyun的pip加速

· 各个节点设置pip加速

mkdir .pip && tee ~/.pip/pip.conf &lt;&lt;-'EOF'[global]trusted-host = mirrors.aliyun.comindex-url = https://mirrors.aliyun.com/pypi/simpleEOF

注意requests包不能在urllib3之后安装,否则会出错:

pip uninstall urllib3
pip uninstall chardet
pip install requests

(这些命令应该都已经在 setup.sh 中执行过了)

yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools iproute lrzsz tree git
yum-config-manager   --add-repo   https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum install -y docker-ce
yum -y install epel-release
systemctl daemon-reload
systemctl enable docker
systemctl restart docker
yum install -y chrony
systemctl start chronyd
systemctl enable chronyd

03拉取并启动Contrail-Kolla-Ansible-Deployer容器

容器的Nightly builds 可以从这里访问: Docker Hub

https://hub.docker.com/r/opencontrailnightly/contrail-kolla-ansible-deployer/tags

例如:

vim /etc/docker/daemon.json
{ "registry-mirrors" : [ "https://hub-mirror.c.163.com",
    "https://registry.docker-cn.com" ] }
systemctl restart docker

export CAD_IMAGE=opencontrailnightly/contrail-kolla-ansible-deployer:master-latest
docker run -td --net host --name contrail_kolla_ansible_deployer $CAD_IMAGE

04 将配置文件复制到容器

instance.yaml:用于配置Tungsten Fabric集群的模板文件。

https://github.com/Juniper/contrail-ansible-deployer/wiki/Contrail-with-Openstack-Kolla#13-configure-necessary-parameters-configinstancesyaml-under-appropriate-parameters

要获得有关如何配置该文件中所有可用参数的信息,可阅读这里:

https://github.com/Juniper/contrail-ansible-deployer/blob/master/README.md#configuration

cat << EOF > instances.yaml
provider_config:  bms:    ssh_pwd: vagrant    ssh_user: root    ntpserver: ntp1.aliyun.com    domainsuffix: localinstances:  tf:    provider: bms    ip: 192.16.35.111    roles:      config_database:      config:      control:      analytics_database:      analytics:      webui:  ops:    provider: bms    ip: 192.16.35.112    roles:
      openstack:
      openstack_compute:  
      vrouter:
        PHYSICAL_INTERFACE: enp0s8
  k8s:    provider: bms    ip: 192.16.35.113    roles:
      k8s_master:
      k8s_node:
      kubemanager:
      vrouter:
        PHYSICAL_INTERFACE: enp0s8
  k01:    provider: bms    ip: 192.16.35.114    roles:
      openstack_compute:
      k8s_node:
      vrouter:
        PHYSICAL_INTERFACE: enp0s8
contrail_configuration:  AUTH_MODE: keystone  KEYSTONE_AUTH_URL_VERSION: /v3
  KEYSTONE_AUTH_ADMIN_PASSWORD: vagrant
  CLOUD_ORCHESTRATOR: openstack
  CONTRAIL_VERSION: latest
  UPGRADE_KERNEL: true
  ENCAP_PRIORITY: "VXLAN,MPLSoUDP,MPLSoGRE"
  PHYSICAL_INTERFACE: enp0s8
global_configuration:
  CONTAINER_REGISTRY: opencontrailnightly
kolla_config:
  kolla_globals:    enable_haproxy: no    enable_ironic: "no"    enable_swift: "no"
    network_interface: "enp0s8"
  kolla_passwords:    keystone_admin_password: vagrant

EOF

export INSTANCES_FILE=instances.yaml
docker cp $INSTANCES_FILE contrail_kolla_ansible_deployer:/root/contrail-ansible-deployer/config/instances.yaml

05准备好所有节点的环境

除了deployer,我在所有节点上都做了一遍。

正常的做法是建个自己的repository放各种image,实验环境节点少,直接国内下载也很快的。

注意python和python-py这两个包是冲突的,只能安装其中之一,最好先全卸载,再安装其中一个:

pip uninstall docker-py docker
 pip install python

yum -y install python-devel python-subprocess32 python-setuptools python-pip

 pip install --upgrade pip

 find / -name *subpro*.egg-info
 find / -name *subpro*.egg-info |xargs rm -rf

pip install -I sixpip install -I docker-compose

将k8s repository改成阿里的,缺省的Google源太慢或不通:vi

playbooks/roles/k8s/tasks/RedHat.yml

yum_repository:
name: Kubernetes
description: k8s repo
baseurl: https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
gpgkey: https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
repo_gpgcheck: yes
gpgcheck: yes
when: k8s_package_version is defined

playbook中安装这些需要访问海外网站,可以从国内下载,然后改个tag:

k8s.gcr.io/kube-apiserver:v1.14.8
k8s.gcr.io/kube-controller-manager:v1.14.8
k8s.gcr.io/kube-scheduler:v1.14.8
k8s.gcr.io/kube-proxy:v1.14.8
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

换个方法变通处理一下

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
docker pull coredns/coredns:1.3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3

再重新给下载的打个tag

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8 k8s.gcr.io/kube-apiserver:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8 k8s.gcr.io/kube-controller-manager:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8 k8s.gcr.io/kube-scheduler:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8 k8s.gcr.io/kube-proxy:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3  k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3

06启动deployer容器,进入其中进行部署

docker start contrail_kolla_ansible_deployer

进入deployer容器:

docker exec -it contrail_kolla_ansible_deployer bashcd /root/contrail-ansible-deployer
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/provision_instances.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/configure_instances.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_openstack.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_k8s.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_contrail.yml

kubectl taint nodes k8s node-role.kubernetes.io/master-

最后一次kubelet升级到最新,遇到CSI的bug,修改一下配置文件后重启kubelet即可:

After experiencing the same issue, editing /var/lib/kubelet/config.yaml to add:
featureGates:  CSIMigration: false

07安装完成后,建2个VM和容器测试一下

yum install -y gcc python-devel
pip install python-openstackclient
pip install python-ironicclient

source /etc/kolla/kolla-toolbox/admin-openrc.sh

如果openstack命令有如下“queue”的报错,是需要python3:

File "/usr/lib/python2.7/site-packages/openstack/utils.py", line 13, in <module>
    import queue
ImportError: No module named queue

rm -f /usr/bin/python
ln -s /usr/bin/python3 /usr/bin/python
pip install python-openstackclient
pip install python-ironicclient
yum install -y python3-pip

yum install -y gcc python-devel wgetpip install --upgrade setuptoolspip install --ignore-installed python-openstackclient

我每次都需要python3,所以干脆也安装了这个:
pip3 install python-openstackclient -i https://mirrors.aliyun.com/pypi/simple/
pip3 install python-ironicclient -i https://mirrors.aliyun.com/pypi/simple/

进入Tungsten Fabric,用浏览器:https://192.16.35.111:8143

进入openstack,用浏览器:https://192.16.35.112

在k8s master上(192.16.35.113):

scp root@192.16.35.114:/opt/cni/bin/contrail-k8s-cni /opt/cni/bin/
mkdir /etc/cni/net.d
scp root@192.16.35.114:/etc/cni/net.d/10-contrail.conf /etc/cni/net.d/10-contrail.conf

wget

https://github.com/cirros-dev/cirros/releases/download/0.4.0/cirros-0.4.0-x86_64-disk.img

官方下载地址

https://download.cirros-cloud.net/

curl -O

https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

wget

http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

wget

http://download.cirros-cloud.net/daily/20161201/cirros-d161201-x86_64-disk.img

(都没有找到带tcpdump的版本)

reboot

source /etc/kolla/kolla-toolbox/admin-openrc.sh

openstack image create cirros --disk-format qcow2 --public --container-format bare --file cirros-0.4.0-x86_64-disk.imgnova flavor-create m1.tiny auto 512 1 1openstack network create net1openstack subnet create --subnet-range 10.1.1.0/24 --network net1 mysubnet1
NET_ID=`openstack network list | grep net1 | awk -F '|' '{print $2}' | tr -d ' '`

nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM1
nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM2

进入k8s_master, 192.16.35.113:

yum install -y git
git clone https://github.com/virtualhops/k8s-demo
kubectl create -f k8s-demo/po-ubuntuapp.yml
kubectl create -f k8s-demo/rc-frontend.yml
kubectl expose rc/frontend
kubectl exec -it ubuntuapp curl frontend # many times

参考方案:

https://github.com/Juniper/contrail-ansible-deployer/wiki/%5B-Container-Workflow%5D-Deploying-Contrail-with-OpenStack

推荐阅读

Tungsten Fabric实战:对接vMX虚拟路由平台填坑
Tungsten Fabric实战:基于K8s的部署踩坑
TF实战Q&A丨你不理解透,出了问题都不知道怎么弄
TF 实战Q&A丨只在此网中,云深不知处


TF实战丨使用Vagrant安装Tungsten Fabric
TF实战丨使用Vagrant安装Tungsten Fabric

点赞
收藏
评论区
推荐文章
blmius blmius
3年前
MySQL:[Err] 1292 - Incorrect datetime value: ‘0000-00-00 00:00:00‘ for column ‘CREATE_TIME‘ at row 1
文章目录问题用navicat导入数据时,报错:原因这是因为当前的MySQL不支持datetime为0的情况。解决修改sql\mode:sql\mode:SQLMode定义了MySQL应支持的SQL语法、数据校验等,这样可以更容易地在不同的环境中使用MySQL。全局s
皕杰报表之UUID
​在我们用皕杰报表工具设计填报报表时,如何在新增行里自动增加id呢?能新增整数排序id吗?目前可以在新增行里自动增加id,但只能用uuid函数增加UUID编码,不能新增整数排序id。uuid函数说明:获取一个UUID,可以在填报表中用来创建数据ID语法:uuid()或uuid(sep)参数说明:sep布尔值,生成的uuid中是否包含分隔符'',缺省为
Jacquelyn38 Jacquelyn38
3年前
2020年前端实用代码段,为你的工作保驾护航
有空的时候,自己总结了几个代码段,在开发中也经常使用,谢谢。1、使用解构获取json数据let jsonData  id: 1,status: "OK",data: 'a', 'b';let  id, status, data: number   jsonData;console.log(id, status, number )
Stella981 Stella981
3年前
KVM调整cpu和内存
一.修改kvm虚拟机的配置1、virsheditcentos7找到“memory”和“vcpu”标签,将<namecentos7</name<uuid2220a6d1a36a4fbb8523e078b3dfe795</uuid
Wesley13 Wesley13
3年前
mysql设置时区
mysql设置时区mysql\_query("SETtime\_zone'8:00'")ordie('时区设置失败,请联系管理员!');中国在东8区所以加8方法二:selectcount(user\_id)asdevice,CONVERT\_TZ(FROM\_UNIXTIME(reg\_time),'08:00','0
Wesley13 Wesley13
3年前
Java日期时间API系列36
  十二时辰,古代劳动人民把一昼夜划分成十二个时段,每一个时段叫一个时辰。二十四小时和十二时辰对照表:时辰时间24时制子时深夜11:00凌晨01:0023:0001:00丑时上午01:00上午03:0001:0003:00寅时上午03:00上午0
Wesley13 Wesley13
3年前
00:Java简单了解
浅谈Java之概述Java是SUN(StanfordUniversityNetwork),斯坦福大学网络公司)1995年推出的一门高级编程语言。Java是一种面向Internet的编程语言。随着Java技术在web方面的不断成熟,已经成为Web应用程序的首选开发语言。Java是简单易学,完全面向对象,安全可靠,与平台无关的编程语言。
Stella981 Stella981
3年前
Django中Admin中的一些参数配置
设置在列表中显示的字段,id为django模型默认的主键list_display('id','name','sex','profession','email','qq','phone','status','create_time')设置在列表可编辑字段list_editable
Wesley13 Wesley13
3年前
MySQL部分从库上面因为大量的临时表tmp_table造成慢查询
背景描述Time:20190124T00:08:14.70572408:00User@Host:@Id:Schema:sentrymetaLast_errno:0Killed:0Query_time:0.315758Lock_
Python进阶者 Python进阶者
10个月前
Excel中这日期老是出来00:00:00,怎么用Pandas把这个去除
大家好,我是皮皮。一、前言前几天在Python白银交流群【上海新年人】问了一个Pandas数据筛选的问题。问题如下:这日期老是出来00:00:00,怎么把这个去除。二、实现过程后来【论草莓如何成为冻干莓】给了一个思路和代码如下:pd.toexcel之前把这
Easter79
Easter79
Lv1
今生可爱与温柔,每一样都不能少。
文章
2.8k
粉丝
5
获赞
1.2k