2024-Q2-notes

MQ

topic操作

import requests
import json
session = requests.Session()
payload = {
    'username': 'admin',
    'password': 'xxxxxxxx'
}
url = 'http://172.24.1.226:9801'
login_url = f'{url}/login/login.do'
response = session.post(login_url, data=payload)
print(response.text)
if response.status_code == 200:
    print('登录成功！')
else:
    print('登录失败！')

list_url = f'{url}/topic/list.query'
res = session.get(list_url)
print(res.text)

print(json.loads(res.text))
tL = json.loads(res.text).get("data").get("topicList")
for item in tL:
    if item.startswith("%"):continue
    print(item)
    # 获取topic 信息
    getUrl = f'{url}/topic/examineTopicConfig.query'
    tp = {"topic": item}
    res = session.get(getUrl, params=tp)
    d = json.loads(res.text)
    for item in d.get("data"):
        print(item.get("readQueueNums"))
        print(item.get("readQueueNums"))

a = {"writeQueueNums":16,"readQueueNums":16,"perm":6,"topicName":"test2","clusterNameList":["DefaultCluster"]}

pUrl = f"{url}/topic/createOrUpdate.do"
r = session.post(pUrl,json=a)
print(r.text, "update ")

Ldap

服务器重启ldap启动异常
解决办法
- 修改hosts文件注释掉多余的配置

Elasticsearch

一、基于快照备份恢复

#!/bin/bash

function base(){
        es_address="172.28.19.148:9200"
        #快照存储位置
        repo_path="/data/elasticsearch/es_snapshots"

        #仓库前缀/后缀
        repo_prefix="es_snapshots"
        repo_suffix=`date +%F`

        #仓库名字
        repo_name="es_snapshots_`date +%F`"

        #快照前缀/后缀
        snap_prefix="snapshot"
        snap_suffix=`date +%F`

        #快照名字
        snap_name="${snap_prefix}_${snap_suffix}"

        curl ${es_address}/_cat/indices
}



function msgs(){
        if [ $? -eq 0 ];then
             echo "$1"
        else
             echo "$2"
        fi
}


function createRepo(){
        echo "createRepo --> ${repo_name}"
        curl -X PUT http://${es_address}/_snapshot/${repo_name} -H 'Content-Type: application/json' -d '{"type": "fs","settings": {"location": "'"${repo_path}/${repo_name}"'","compress": true}}' 2>/dev/null
        echo ""
}



function createSnap(){
        echo "createSnap --> ${snap_name}"
        curl -X PUT http://${es_address}/_snapshot/${repo_name}/${snap_name}?wait_for_completion=true
        if [ $? -eq 0 ];then
                cd ${repo_path}
                tar -zcvf ${repo_name}.tar.gz ${repo_name} &>/dev/null
                mv ${repo_name} /tmp
                rm -rf /tmp/${repo_name}
        fi

}

#根具索引创建快照
function createOneIndexSnap(){
        index_name="filebeat-6.1.4-2022.08.26"
        curl -X PUT http://${es_address}/_snapshot/${repo_name}/${index_name}?wait_for_completion=true -H 'Content-Type: application/json' -d '{"indices": "'"${index_name}"'","ignore_indices": "missing"}' 2>/dev/null
        echo ""

}

function delSnap(){
        curl -X DELETE http://${es_address}/_snapshot/${repo_name}
        msgs "删除snap 成功" "删除snap 失败"
        echo ""
}


function restoreSnap(){
        #全量恢复
        #curl -X POST "http://${es_address}/_snapshot/${repo_name}/${snap_name}/_restore?wait_for_completion=true"

        index_name="rabbit_audience_index_47_bak"

        #从全量中恢复指定索引
        curl -X POST "http://${es_address}/_snapshot/${repo_name}/${snap_name}/_restore?wait_for_completion=true" -H 'Content-Type: application/json' -d '{"indices": "'"${index_name}"'","ignore_unavailable": true}' 2>/dev/null

        #指定索引
        #curl -X POST http://${es_address}/_snapshot/${repo_name}/${index_name}/_restore?wait_for_completion=true -H 'Content-Type: application/json' -d '{"indices": "'"${index_name}"'","ignore_unavailable": true}'
        echo ""
}


#定时删除5天之前的快照
function cronDel(){
        day=`date -d "5 day ago" +%F`
        #day=`date +%F`
        repo_name="${repo_prefix}_${day}"
        curl -X DELETE http://${es_address}/_snapshot/${repo_name}/
        rm -f "${repo_path}/${repo_name}.tar.gz"
        echo ""
}



function main(){
        echo "start --> `date +%F-%T`"
        base
        createRepo
        createSnap
        cronDel
        echo "stop --> `date +%F-%T`"
        #delSnap
        #createOneIndexSnap
        #restoreSnap
}


main

二、使用nfs作为快照目录

2.1 踩坑权限问题

报没权限,集群中使用的是es用户启动。每个节点es用户的uid、gid不一样，导致创建快照的时候没有权限。

2.2 解决办法

在各个节点创建一个统一的uid/gid用户。示例命令如下

groupadd -g 1020 elastic ; useradd -u 1020 -g elastic elastic

将es用户加入到elastic组中

usermod -G elastic es

修改nfs的配置主要是uid/gid的值

/data3 *(rw,sync,all_squash,anonuid=1020,anongid=1020)

修改完配置之后重启nfs. es 集群重新创建快照即可

三、es索引问题

3.1 索引只读

索引无法操作，磁盘空间足够

2. 解决方法: 修改索引配置

curl -X PUT 172.28.19.148:9200/_all/_settings -H 'Content-Type: application/json' -d '{"index": { "blocks": {"read_only_allow_delete": "false"}}}'

私仓repo

lvm 磁盘扩容

公有云新增磁盘
新买的磁盘分区

fdisk /dev/新盘

扩容pv

pvcreate /dev/vdd1

扩容vg

vgextend data /dev/vdd1

扩容lv

lvextend -l +100%FREE /dev/data/data

扩容文件系统

resize2fs -p -F /dev/mapper/data-data

mysql

一、数据转换

mysql8.0 转 mysql5.7

sed -i -e "s#utf8mb4_0900_ai_ci#utf8_general_ci#g" -e "s#utf8_croatian_ci#utf8_general_ci#" -e "s#utf8mb4_general_ci#utf8_general_ci#" -e "s#utf8mb4#utf8#" -e "s#utf8mb4_bin#utf8_general_ci#" -e 's#utf8mb4_unicode_ci#utf8_general_ci#g' *

Jenkins

jenkins对接ldap开启权限管理
- jenkins安装插件: Matrix Authorization Strategy (授权), LDAP Plugin
- 系统管理 --> 全局安全配置 --> 安全域

ocserv 部署

配置文件

auth = "plain[/etc/ocserv/ocpasswd]"
tcp-port = 443
udp-port = 443
run-as-user = nobody
run-as-group = nogroup
socket-file = /var/run/ocserv-socket
server-cert = /etc/ocserv/server-cert.pem
server-key = /etc/ocserv/server-key.pem
max-clients = 16
max-same-clients = 2
keepalive = 32400
dpd = 90
mobile-dpd = 1800
cert-user-oid = 2.5.4.3

auth-timeout = 240
idle-timeout = 86400
mobile-idle-timeout = 86400
min-reauth-time = 300
max-ban-score = 80
ban-reset-time = 1200
cookie-timeout = 300
deny-roaming = false
rekey-time = 172800
rekey-method = ssl
use-occtl = true
pid-file = /var/run/ocserv.pid
net-priority = 6
device = vpns
predictable-ips = true
default-domain = xfanyi.top

ipv4-network = 192.168.1.0
ipv4-netmask = 255.255.255.0

route = default

dns = 1.1.1.1
dns = 223.5.5.5

启动命令

docker run -d --name ocserv \
    -p 7443:443/tcp \
    -p 7443:443/udp \
    -v /data/ocserv/ocserv.conf:/etc/ocserv/ocserv.conf \
    -v /data/script/keys_and_certs/9d3859fed4722d2e.xfanyi.top.crt:/etc/ocserv/server-cert.pem \
    -v /data/script/keys_and_certs/9d3859fed4722d2e.xfanyi.top.key:/etc/ocserv/server-key.pem \
    -v /data/ocserv/ocpasswd:/etc/ocserv/ocpasswd \
    tommylau/ocserv

增加用户

ocpasswd -c /tmp/auth_file xfy

下载软件包依赖

repotrack -p /path/to/download/ httpd

2024-Q2-notes

2024-Q2-notes

MQ

topic操作

Ldap

Elasticsearch

一、基于快照备份恢复

二、使用nfs作为快照目录

2.1 踩坑权限问题

2.2 解决办法

三、es索引问题

3.1 索引只读

私仓repo

lvm 磁盘扩容

mysql

一、数据转换

Jenkins

ocserv 部署

配置文件

启动命令

增加用户

下载软件包依赖

文章目录

评论

2024-Q2-notes

2024-Q2-notes

MQ

topic操作

Ldap

Elasticsearch

一、基于快照备份恢复

二、使用nfs作为快照目录

2.1 踩坑 权限问题

2.2 解决办法

三、es索引问题

3.1 索引只读

私仓repo

lvm 磁盘扩容

mysql

一、数据转换

Jenkins

ocserv 部署

配置文件

启动命令

增加用户

下载软件包依赖

文章目录

评论

2.1 踩坑权限问题