kk的blog

2023/02/22

昨天有朋友問我說proxmox的guest開不了几

不知是什麼問題

連進去看了一下

發現一個guest開了好几個HD

而且每個HD都開到2T

而且還做了好几個snapshot

導致實際上guest的HD 膨脹到5T或更大

把空間全部吃滿了

所以無法開几

因為是使用qcow2格式

所以snapshot會長在原來的檔案上

我是覺得奇怪

guest開那麼多個2T的檔案

user不會覺得效能不好嗎

解決的方式就是把舊的snapshot砍一砍

再觀察看看

2023/02/13

今天早上要進ntop管理介面的時候

打完帳號密碼登不進去

進os看了一下HD滿了

然後再看log

出現一堆如下的訊息把HD塞爆了

Feb 13 08:14:29 W-ntopng ntopng[286]: 13/Feb/2023 08:14:29 [SQLiteAlertStore.cpp:151] ERROR: SQL Error: database disk image is malformed

Feb 13 08:14:29 W-ntopng ntopng[286]: INSERT INTO flow_alerts (alert_id, interface_id, tstamp, tstamp_end, severity, ip_version, cli_ip, srv_ip, cli_port, srv_port, vlan_id, is_cli_attacker, is_cli_victim, is_srv_attacker, is_srv_victim, proto, l7_proto, l7_master_proto, l7_cat, cli_name, srv_name, cli_country, srv_country, cli_blacklisted, srv_blacklisted, cli_location, srv_location, cli2srv_bytes, srv2cli_bytes, cli2srv_pkts, srv2cli_pkts, first_seen, community_id, score, flow_risk_bitmap, alerts_map, cli_host_pool_id, srv_host_pool_id, cli_network, srv_network, probe_ip, input_snmp, output_snmp, json, info) VALUES (26, 3, 1676247257, 1676247266, 3, 4, '192.168.40.66', '192.168.0.65', 44983, 80, 0, 0, 0, 0, 0, 6, 7, 0, 5, '', '', '', '', 0, 0, 0, 0, 126, 120, 2, 2, 1676247257, '1:rj5vzKw7WQX8TONTQ++bh3BkBh8=', 10, 70368744177664, X'04000000', 0, 0, 65535, 65535, '0.0.0.0', 0, 0, '{"ntopng.key":12345678,"hash_entry_id":23456789,"alert_generation": {"script_key":"ndpi_unidirectional_traffic","subdir":"flow","flow_risk_info":"{\"46\":\"No client to server traffic\"}"},"proto": {"http": {},"confidence":0}}', '');

google了一下是 sqllite 因為斷電導致有問題

果然

斷一次電事情一堆

看是有recovery sqlite的方法

算了

直接倒回事發前一天晚上的備分好了

倒回後目前正常

再觀察看看

https://blog.csdn.net/wolfking0608/article/details/71076588

2023/02/11

今天下午几房斷電

有一台graylog啟動後

三個 service都有起來

但從管理介面看log全都卡住

過了一個小時還是沒有消化

想說應該是 elasticsearch 有問題了

看了log

[2023-02-11T20:27:56,520][WARN ][o.e.c.r.a.AllocationService] [localhost.localdomain] failing shard [failed shard, shard [graylog_666][2], node[0l7asmrIRFeIxc3FyAB14Q], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=yqeR9a7CSUC4ZIIz-a07Gw], unassigned_info[[reason=ALLOCATION_FAILED], at[2023-02-11T12:27:55.997Z], failed_attempts[4], failed_nodes[[0l7asmrIRFeIxc3FyAB14Q]], delayed=false, details[failed shard on node [0l7asmrIRFeIxc3FyAB14Q]: failed recovery, failure RecoveryFailedException[[graylog_666][2]: Recovery failed on {localhost.localdomain}{0l7asmrIRFeIxc3FyAB14Q}{AHesmcGhQvGWAw7Gxl2V6A}{10.10.0.1}{10.10.01:9300}{dimr}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/mnt/elasticsearch/nodes/0/indices/soJ39cmwT5-UlEyVIPvfAg/2/index/_x63f.fdt]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[graylog_666][2]: Recovery failed on {localhost.localdomain}{0l7asmrIRFeIxc3FyAB14Q}{AHesmcGhQvGWAw7Gxl2V6A}{10.10.0.1}{10.10.0.1:9300}{dimr}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/mnt/elasticsearch/nodes/0/indices/soJ39cmwT5-UlEyVIPvfAg/2/index/_x63f.fdt]; ], markAsStale [true]]

org.elasticsearch.indices.recovery.RecoveryFailedException: [graylog_666][2]: Recovery failed on {localhost.localdomain}{0l7asmrIRFeIxc3FyAB14Q}{AHesmcGhQvGWAw7Gxl2V6A}{10.10.0.1}{10.10.0.1:9300}{dimr}

果然

手動 rotate active write index

有消化了

再觀察看看

2023/02/08

nftables 雖然是很久的東西了還是記一下

安裝

dnf install nftables

清空所有

nft flush ruleset

設定新table

nft add table inet filter

新增一個chain 並預設規則

nft add chain inet filter INPUT { type filter hook input priority 0 \; counter \; policy accept \; }

在chain加上新規則

nft insert rule inet filter INPUT ip saddr 192.168.12.85 tcp dport 22 drop

列出規則

nft list table inet filter

列出規則顯示 handle 號以利刪除

nft -an list table inet filter

列出所有table的規則

nft -an list ruleset

刪除規則

nft delete rule inet filter INPUT handle 2

nft加的規則 iptables 去看會不完整

但是有作用的

下完 nft flush ruleset 後

再下 nft -an list ruleset 是看不到資料的

但如果再下iptables -L

下完後

再 nft -an list ruleset

就會看到如下的ruleset

table ip filter { # handle 3

chain INPUT { # handle 1

type filter hook input priority 0; policy accept;

}

chain FORWARD { # handle 2

type filter hook forward priority 0; policy accept;

}

chain OUTPUT { # handle 3

type filter hook output priority 0; policy accept;

}

預設所有下的指令重開几就會清掉如果要重開几自動執行

nft list ruleset >> /etc/sysconfig/nftables.conf

或

先匯出成檔案

nft list ruleset > /etc/nftables/nft_policy.nft

然後再 /etc/sysconfig/nftables.conf

include "/etc/nftables/nft_policy.nft"

重開後撈進來

如果確定不再需要 iptables

移除

dnf remove iptables

https://www.cnblogs.com/vincenshen/p/12333904.html

https://www.linuxprobe.com/linux-nftables-iptables.html

2023/02/07

sshpass

範例

sshpass -p passwd ssh root@10.0.0.1 date

2023/02/04

今天在檢查 pbs 時又出現 GC (garbage collection) warning 的log

再去看前一天的備分

是顯示備份完成且沒有錯誤的

不知道為什麼備分完成沒有錯誤但GC時會出現如下的錯誤

2023-02-02T00:00:22+08:00: WARN: warning: unable to access non-existent chunk 4d9f87572f2ff8d9f324aef1263e1ab47181a764aac801918b6dd5567fdfdde9, required by "/mnt/nfs418/ct/112/2023-01-31T16:00:47Z/catalog.pcat1.didx"

2023-02-02T00:00:22+08:00: WARN: warning: unable to access non-existent chunk e82ad3ac9b4b29c55420a44c29029c1a69ebd2cae156994c7e6a4f6a3b44524d, required by "/mnt/nfs418/ct/112/2023-01-31T16:00:47Z/catalog.pcat1.didx"

2023-02-02T00:00:22+08:00: WARN: warning: unable to access non-existent chunk 7eeadcfafebe86f0244ab4b07167644784be8485da119208d91e078efb48a7de, required by "/mnt/nfs418/ct/112/2023-01-31T16:00:47Z/root.pxar.didx"

而且問題來了

再接下來每一次備分都會顯示備分完成無異常

但GC就會一直錯誤

而且一旦GC有錯誤這個備分就無法還原

目前pbs在GC有錯誤時無法主動發mail 告警

所以解決方法就是寫個程式每天檢查GC是否有錯誤

如果有

就要把有錯誤相關的備分砍了

讓接下來的備分能正常

2023/02/03

atftp

sudo apt install atftpd

mkdir /tftp_data

chmod -R 777 /tftp_data

atftpd --daemon --port 69 /tftp_data

2023/01/10

前一陣子發生了一件很OX的事

試了很久現在做個記錄

因為有幫別的部門架了一台proxmox

而且那個部門也有一台nas

因此我就在那台nas上開了nfs然後用pbs備份

他老兄在某一天竟然進到nas把我備分用的那個nfs裡的資料砍了

然後第二天就出現了如下的error

ProxmoxBackup Server 2.3-2

2022-12-29T00:00:00+08:00: starting garbage collection on store nfs418

2022-12-29T00:00:00+08:00: task triggered by schedule 'daily'

2022-12-29T00:00:00+08:00: Start GC phase1 (mark used chunks)

2022-12-29T00:01:03+08:00: marked 5% (1 of 17 index files)

2022-12-29T00:04:14+08:00: marked 11% (2 of 17 index files)

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk c3fe251560dcd2cc5aef7cfbd6669d0dd9ca7491c455f537efc6d319b09892ec, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 6dc29793341c20c7c80910a73893501b034a0e29c14a444d214d835ccffc0d16, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 1f81c244f9b7816ab37d3ef7ffdcd10443eb1bddb3fa44e036186b73f1fee33a, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 95c7e6747d43b5b516e1768a1f258f352aeb47b23fd46575440dc3d820d1b253, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 9513a7e5c650245d3344115115b21285cc8f426e5284fa04159ea96e49856535, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 68b2b40dd3bacba9e649c67862011934bc7f048d8ea2d35fbd95c39f9d5cf7c4, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 494608d49f57039fd7a8378e1a2a2cf6bb0688773a525addc2f54507a94cd11d, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 09eab384d5f3cbf657db1c0750ef52be2f1dbdf134f977a24dc095382b6e25ed, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 9c1208d43242276c75a4f65f41054e4bc0e7e1144a61c951902bb8df94f055c9, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"

..........................................

後面還一大堆

本來是想說把pbs上的datastore砍了重建應該可以解決問題

forum上也是醬說的

結果不是

我砍了datastore重建

甚至在nas上再開另一個nfs後再重建datastore

過沒几天又會出現如上的錯誤

最後的解決方法是直接重裝一台新的pbs

問題才解決

無言中

2023/01/07

在一台几器上使用mutt寄信時出現

SMTP session failed: 502 5.5.2 Error: command not recognized

到mail server上看log

Jan 7 05:39:01 mail postfix/smtpd[1874477]: improper command pipelining after EHLO from unknown[10.0.01]: hostabc\r\n

感覺好像是hostname的問題

於是在 .muttrc 裡加上一行

set hostname=myhost

就ok了

2022/12/29

最近換了新環控系統

因為是跑在win上資料放在 mssql

所以要去把告警資料撈出來用line傳送

在linux上要先安裝 sqlcmd

m$官方有提供安裝文件如下

安裝完成後會放在

/opt/mssql-tools/bin/

連線語法如下

/opt/mssql-tools/bin/sqlcmd -S 10.1.2.3 -U sa -P 'passwd' -d dbname

連線成功會出現

指令下完要記得再下 go 才能執行

一些基本指令

顯示所有資料庫

SELECT Name from sys.databases

使用資料庫

use daname

顯示db的所有資料表

SELECT * FROM Information_Schema.TABLES

顯示table schema

sp_help tablename

只顯示前 n 筆資料

select top n * from alarmhistory

找出最近時間的資料

select max(time_col) from table

備分資料庫

只能備到server本几的HD或UNC 路徑

sqlcmd -S 10.1.2.3 -U sa -P 'password' -Q "BACKUP DATABASE dbname TO DISK = '\\%tmp%\Report.bak' WITH NOFORMAT, NOINIT, NAME = 'dbname-full', SKIP, NOREWIND, NOUNLOAD, STATS = 10"

https://learn.microsoft.com/zh-tw/sql/linux/sql-server-linux-setup-tools?view=sql-server-ver16#RHEL

2022/12/27

今天有人問個問題

需要創建大量目錄

但目錄名稱有簡体字

本來想說應該沒什麼

結果一跑才知道在dos模式utf-8顯示會有問題

而無法建立

所以批次檔要加上

chcp 65001

才行

範例如下

%echo off

chcp 65001

for /f %%a in (data) do (

echo %%a

md %%a

)

https://tomkuo139.blogspot.com/2017/04/widnows-command-bat-utf8.html

2022/12/23

https://zhangguanzhang.github.io/2022/02/18/proxmox-boot-disk-lvmid-not-found/#/%E8%B5%B7%E5%9B%A0

2022/12/21

這几天試了一下 mosh

主要是在網路環境不佳時

使用ssh可以減少lag的情況

尤其是在4G或wifi時

在不特定 client ip 的情況下建議使用 ssh key認証

只要在server 端安裝 mosh 就可以

apt install mosh

client 端連線要選擇支援mosh的軟体

mobaxterm

JuiceSSH

這二個都有支援

client端發起連線就會自動呼叫 server端的 mosh

所以server端不用起daemon或事先執行

https://mosh.org/

2022/12/20

今天更新 debian

直接下完

apt update

apt upgrade -y 後

librenms就打不開了

跑了 ./daily.sh

出現

Updating to latest codebase OK

Updating Composer packages FAIL

> LibreNMS\ComposerHelper::preInstall

Installing dependencies from lock file

Verifying lock file contents can be installed on current platform.

Your lock file does not contain a compatible set of packages. Please run composer update.

Problem 1

- fgrosse/phpasn1 is locked to version v2.4.0 and an update of this package was not requested.

- fgrosse/phpasn1 v2.4.0 requires php ~7.1.0 || ~7.2.0 || ~7.3.0 || ~7.4.0 || ~8.0.0 || ~8.1.0 -> your php version (8.2.0) does not satisfy that requirement.

Problem 2

- fgrosse/phpasn1 v2.4.0 requires php ~7.1.0 || ~7.2.0 || ~7.3.0 || ~7.4.0 || ~8.0.0 || ~8.1.0 -> your php version (8.2.0) does not satisfy that requirement.

- web-token/jwt-core v2.2.11 requires fgrosse/phpasn1 ^2.0 -> satisfiable by fgrosse/phpasn1[v2.4.0].

- web-token/jwt-core is locked to version v2.2.11 and an update of this package was not requested.

Updated from ae22662c8 to 1ab238a4d OK

Updating SQL-Schema OK

Updating submodules OK

Cleaning up DB OK

Fetching notifications OK

Caching PeeringDB data OK

Caching Mac OUI data OK

看來是php不能升到 8.2

只能倒回備分

再排除以下的更新

php-cli/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-common/bullseye 2:93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:93+0~20221029.44+debian11~1.gbpaae3e9]

php-curl/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-fpm/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-gd/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-gmp/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-json/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-mbstring/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-mysql/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-pear/bullseye 1:1.10.13+submodules+notgz+2022032202-2+0~20221209.38+debian11~1.gbpfd4c1d all [upgradable from: 1:1.10.13+submodules+notgz+2022032202-2+0~20220330.37+debian11~1.gbpfd4c1d]

php-snmp/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-xml/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

php-zip/bullseye 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [upgradable from: 2:8.1+93+0~20221029.44+debian11~1.gbpaae3e9]

再來把以上的package hold住不要更新

指令如下

Hold a package:

sudo apt-mark hold <package-name>

Remove the hold:

sudo apt-mark unhold <package-name>

Show all packages on hold:

sudo apt-mark showhold

指令下完後 apt upgrade 時就不會更新

不過如果是下 apt install 還是會更新此時就不受 hold影響要特別注意

https://askubuntu.com/questions/18654/how-to-prevent-updating-of-a-specific-package

2022/12/16

graylog 迎來了5 版的更新

依然還是無法原几升級

記錄一下安裝的過程以debian 11 為例

依照官方文件的流程先裝mongodb

由於新版的mongodb需要使用 cpu 的 avx 功能

所以proxmox 上 guest的 cpu type 必須要改成 host 不能使用預設值 kvm64

光是這個問題就卡關好久

而且mongodb 5版裝完後無法執行 6 版沒問題

接下來新的版本可以選擇使用

OpenSearch 或 Elasticsearch

我是選擇 Elasticsearch

第三步是安裝 graylog server

安裝後調整參數

/etc/elasticsearch/jvm.options 這個檔

-Xms8g

-Xmx8g

以上的參數依照記憶体進行調整

/etc/default/graylog-server 檔案裡

GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow"

-Xms1g -Xmx1g

這二個參數也是依照記憶体進行調整

其他依文件安裝問題不大

不過文件上有些斷行有問題需要自行調整

例如

wget -qO - https://www.mongodb.org/static/pgp/server-5.x.asc | sudo apt-key add -

echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/5.x main" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.x.list

及

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

另外使用almalinux 8 跟 9 安裝過程沒有出現錯誤但 graylog server 啟動打開管理介面

出現以下的錯誤

Error Loading active authentication service type failed with status: SyntaxError: JSON.parse: unexpected character at line 2 column 1 of the JSON data

不知道是那裡出問題有空再來試了

找到問題了 firewalld 沒關

systemctl disable firewalld

systemctl stop firewalld

https://go2docs.graylog.org/5-0/what_is_graylog/what_is_graylog.htm