有一台僅供內部使用的 proxy 或 server
想要透過 tunnel 使用
架構圖如下
2023/04/20
2023/04/14
為了因應centos 7 EOS
今天在轉移程式到 almalinux 9 時 mutt 一直無法寄信
mail server 的log如下
Apr 14 10:09:09 mail postfix/smtpd[523701]: connect from unknown[10.0.0.1]
Apr 14 10:09:09 mail postfix/smtpd[523701]: lost connection after STARTTLS from unknown[10.0.0.1]
看來almalinux 9的mutt 預設會使用 STARTTLS
在 .muttrc 加上以下這行
set ssl_force_tls = no
目前寄信正常了
20240308 後記
轉換到新mail server後 有些mutt又出現不能寄信的狀況
以下是log
Mar 8 06:13:01 mail postfix/smtpd[27212]: warning: TLS library problem: error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:320:
Mar 8 06:13:01 mail postfix/smtpd[27212]: lost connection after STARTTLS from unknown[10.0.0.1]
必須在.muttrc再加上 set ssl_starttls = no
包括之前說明的共二行如下
set ssl_force_tls = no
set ssl_starttls = no
目前正常了 再觀察看看
2023/04/09
2023/03/02
使用curl 撈取 loki 資料的語法
依需求需要更改之處
job="abc"
查詢的關鍵字 192.168.1.2
查詢的區間
curl -G -s "http://10.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="abc"} |~ "192.168.1.2"' --data-urlencode "start=$(date -u +'%Y-%m-%dT%H:%M:%SZ' -d '-8 hour')" --data-urlencode "end=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
curl -G -s "http://10.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="abc"} |~ "192.168.1.2"' --data-urlencode "start=$(date -u +'%Y-%m-%dT%H:%M:%SZ' -d '-7 day')" --data-urlencode "end=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"|jq
2023/02/28
今天早上三點多開始收到ntopng的告警
進主几看了一下 process不見了
重開也沒用
看了一下log
ntopng果然有更新
更新後起不來
接下來看ntopng的log
發現本次更新後必須使用到 libbpf.so.0
Feb 28 08:09:17 W-ntopng-ubuntu-2004 ntopng[3247]: /usr/bin/ntopng: error while loading shared libraries: libbpf.so.0: cannot open shared object file: No such file or directory
Feb 28 08:09:22 W-ntopng-ubuntu-2004 ntopng[3272]: /usr/bin/ntopng: error while loading shared libraries: libbpf.so.0: cannot open shared object file: No such file or directory
Feb 28 08:09:28 W-ntopng-ubuntu-2004 ntopng[3294]: /usr/bin/ntopng: error while loading shared libraries: libbpf.so.0: cannot open shared object file: No such file or directory
apt install libbpf0
目前正常了 再觀察看看
2023/02/22
2023/02/13
今天早上要進ntop管理介面的時候
打完帳號密碼登不進去
進os看了一下HD滿了
然後再看log
出現一堆如下的訊息 把HD塞爆了
Feb 13 08:14:29 W-ntopng ntopng[286]: 13/Feb/2023 08:14:29 [SQLiteAlertStore.cpp:151] ERROR: SQL Error: database disk image is malformed
Feb 13 08:14:29 W-ntopng ntopng[286]: INSERT INTO flow_alerts (alert_id, interface_id, tstamp, tstamp_end, severity, ip_version, cli_ip, srv_ip, cli_port, srv_port, vlan_id, is_cli_attacker, is_cli_victim, is_srv_attacker, is_srv_victim, proto, l7_proto, l7_master_proto, l7_cat, cli_name, srv_name, cli_country, srv_country, cli_blacklisted, srv_blacklisted, cli_location, srv_location, cli2srv_bytes, srv2cli_bytes, cli2srv_pkts, srv2cli_pkts, first_seen, community_id, score, flow_risk_bitmap, alerts_map, cli_host_pool_id, srv_host_pool_id, cli_network, srv_network, probe_ip, input_snmp, output_snmp, json, info) VALUES (26, 3, 1676247257, 1676247266, 3, 4, '192.168.40.66', '192.168.0.65', 44983, 80, 0, 0, 0, 0, 0, 6, 7, 0, 5, '', '', '', '', 0, 0, 0, 0, 126, 120, 2, 2, 1676247257, '1:rj5vzKw7WQX8TONTQ++bh3BkBh8=', 10, 70368744177664, X'04000000', 0, 0, 65535, 65535, '0.0.0.0', 0, 0, '{"ntopng.key":12345678,"hash_entry_id":23456789,"alert_generation": {"script_key":"ndpi_unidirectional_traffic","subdir":"flow","flow_risk_info":"{\"46\":\"No client to server traffic\"}"},"proto": {"http": {},"confidence":0}}', '');
google了一下是 sqllite 因為斷電導致有問題
果然
斷一次電事情一堆
看是有recovery sqlite的方法
算了
直接倒回事發前一天晚上的備分好了
倒回後目前正常
再觀察看看
https://blog.csdn.net/wolfking0608/article/details/71076588
2023/02/11
今天下午几房斷電
有一台graylog啟動後
三個 service都有起來
但從管理介面看log全都卡住
過了一個小時還是沒有消化
想說應該是 elasticsearch 有問題了
看了log
[2023-02-11T20:27:56,520][WARN ][o.e.c.r.a.AllocationService] [localhost.localdomain] failing shard [failed shard, shard [graylog_666][2], node[0l7asmrIRFeIxc3FyAB14Q], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=yqeR9a7CSUC4ZIIz-a07Gw], unassigned_info[[reason=ALLOCATION_FAILED], at[2023-02-11T12:27:55.997Z], failed_attempts[4], failed_nodes[[0l7asmrIRFeIxc3FyAB14Q]], delayed=false, details[failed shard on node [0l7asmrIRFeIxc3FyAB14Q]: failed recovery, failure RecoveryFailedException[[graylog_666][2]: Recovery failed on {localhost.localdomain}{0l7asmrIRFeIxc3FyAB14Q}{AHesmcGhQvGWAw7Gxl2V6A}{10.10.0.1}{10.10.01:9300}{dimr}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/mnt/elasticsearch/nodes/0/indices/soJ39cmwT5-UlEyVIPvfAg/2/index/_x63f.fdt]; ], allocation_status[deciders_throttled]], message [failed recovery], failure [RecoveryFailedException[[graylog_666][2]: Recovery failed on {localhost.localdomain}{0l7asmrIRFeIxc3FyAB14Q}{AHesmcGhQvGWAw7Gxl2V6A}{10.10.0.1}{10.10.0.1:9300}{dimr}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: NoSuchFileException[/mnt/elasticsearch/nodes/0/indices/soJ39cmwT5-UlEyVIPvfAg/2/index/_x63f.fdt]; ], markAsStale [true]]
org.elasticsearch.indices.recovery.RecoveryFailedException: [graylog_666][2]: Recovery failed on {localhost.localdomain}{0l7asmrIRFeIxc3FyAB14Q}{AHesmcGhQvGWAw7Gxl2V6A}{10.10.0.1}{10.10.0.1:9300}{dimr}
果然
手動 rotate active write index
有消化了
再觀察看看
2023/02/08
nftables 雖然是很久的東西了 還是記一下
安裝
dnf install nftables
清空所有
nft flush ruleset
設定新table
nft add table inet filter
新增一個chain 並預設規則
nft add chain inet filter INPUT { type filter hook input priority 0 \; counter \; policy accept \; }
在chain加上新規則
nft insert rule inet filter INPUT ip saddr 192.168.12.85 tcp dport 22 drop
列出規則
nft list table inet filter
列出規則 顯示 handle 號 以利刪除
nft -an list table inet filter
列出所有table的規則
nft -an list ruleset
刪除規則
nft delete rule inet filter INPUT handle 2
nft加的規則 iptables 去看會不完整
但是有作用的
下完 nft flush ruleset 後
再下 nft -an list ruleset 是看不到資料的
但如果再下iptables -L
下完後
再 nft -an list ruleset
就會看到如下的ruleset
table ip filter { # handle 3
chain INPUT { # handle 1
type filter hook input priority 0; policy accept;
}
chain FORWARD { # handle 2
type filter hook forward priority 0; policy accept;
}
chain OUTPUT { # handle 3
type filter hook output priority 0; policy accept;
}
}
預設所有下的指令 重開几就會清掉 如果要重開几自動執行
nft list ruleset >> /etc/sysconfig/nftables.conf
或
先匯出成檔案
nft list ruleset > /etc/nftables/nft_policy.nft
然後再 /etc/sysconfig/nftables.conf
include "/etc/nftables/nft_policy.nft"
重開後撈進來
如果確定不再需要 iptables
移除
dnf remove iptables
2023/02/07
2023/02/04
今天在檢查 pbs 時又出現 GC (garbage collection) warning 的log
再去看前一天的備分
是顯示備份完成且沒有錯誤的
不知道為什麼備分完成沒有錯誤但GC時會出現如下的錯誤
2023-02-02T00:00:22+08:00: WARN: warning: unable to access non-existent chunk 4d9f87572f2ff8d9f324aef1263e1ab47181a764aac801918b6dd5567fdfdde9, required by "/mnt/nfs418/ct/112/2023-01-31T16:00:47Z/catalog.pcat1.didx"
2023-02-02T00:00:22+08:00: WARN: warning: unable to access non-existent chunk e82ad3ac9b4b29c55420a44c29029c1a69ebd2cae156994c7e6a4f6a3b44524d, required by "/mnt/nfs418/ct/112/2023-01-31T16:00:47Z/catalog.pcat1.didx"
2023-02-02T00:00:22+08:00: WARN: warning: unable to access non-existent chunk 7eeadcfafebe86f0244ab4b07167644784be8485da119208d91e078efb48a7de, required by "/mnt/nfs418/ct/112/2023-01-31T16:00:47Z/root.pxar.didx"
而且問題來了
再接下來每一次備分都會顯示備分完成無異常
但GC就會一直錯誤
而且一旦GC有錯誤這個備分就無法還原
目前pbs在GC有錯誤時無法主動發mail 告警
所以解決方法就是寫個程式每天檢查GC是否有錯誤
如果有
就要把有錯誤相關的備分砍了
讓接下來的備分能正常
2023/02/03
2023/01/10
前一陣子發生了一件很OX的事
試了很久 現在做個記錄
因為有幫別的部門架了一台proxmox
而且那個部門也有一台nas
因此我就在那台nas上開了nfs然後用pbs備份
他老兄在某一天竟然進到nas把我備分用的那個nfs裡的資料砍了
然後第二天就出現了如下的error
ProxmoxBackup Server 2.3-2
2022-12-29T00:00:00+08:00: starting garbage collection on store nfs418
2022-12-29T00:00:00+08:00: task triggered by schedule 'daily'
2022-12-29T00:00:00+08:00: Start GC phase1 (mark used chunks)
2022-12-29T00:01:03+08:00: marked 5% (1 of 17 index files)
2022-12-29T00:04:14+08:00: marked 11% (2 of 17 index files)
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk c3fe251560dcd2cc5aef7cfbd6669d0dd9ca7491c455f537efc6d319b09892ec, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 6dc29793341c20c7c80910a73893501b034a0e29c14a444d214d835ccffc0d16, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 1f81c244f9b7816ab37d3ef7ffdcd10443eb1bddb3fa44e036186b73f1fee33a, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 95c7e6747d43b5b516e1768a1f258f352aeb47b23fd46575440dc3d820d1b253, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 9513a7e5c650245d3344115115b21285cc8f426e5284fa04159ea96e49856535, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 68b2b40dd3bacba9e649c67862011934bc7f048d8ea2d35fbd95c39f9d5cf7c4, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 494608d49f57039fd7a8378e1a2a2cf6bb0688773a525addc2f54507a94cd11d, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 09eab384d5f3cbf657db1c0750ef52be2f1dbdf134f977a24dc095382b6e25ed, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
2022-12-29T00:04:14+08:00: WARN: warning: unable to access non-existent chunk 9c1208d43242276c75a4f65f41054e4bc0e7e1144a61c951902bb8df94f055c9, required by "/mnt/nfs418/vm/108/2022-12-27T15:45:38Z/drive-virtio0.img.fidx"
..........................................
後面還一大堆
本來是想說把pbs上的datastore砍了重建應該可以解決問題
forum上也是醬說的
結果不是
我砍了datastore重建
甚至在nas上再開另一個nfs後再重建datastore
過沒几天又會出現如上的錯誤
最後的解決方法是直接重裝一台新的pbs
問題才解決
無言中
2023/01/07
2022/12/29
最近換了新環控系統
因為是跑在win上資料放在 mssql
所以要去把告警資料撈出來用line傳送
在linux上要先安裝 sqlcmd
m$官方有提供安裝文件如下
安裝完成後會放在
/opt/mssql-tools/bin/
連線語法如下
/opt/mssql-tools/bin/sqlcmd -S 10.1.2.3 -U sa -P 'passwd' -d dbname
連線成功會出現
1>
指令下完要記得 再下 go 才能執行
一些基本指令
顯示所有資料庫
SELECT Name from sys.databases
使用資料庫
use daname
顯示db的所有資料表
SELECT * FROM Information_Schema.TABLES顯示table schema
sp_help tablename
只顯示前 n 筆資料
select top n * from alarmhistory
找出最近時間的資料
select max(time_col) from table
備分資料庫
只能備到server本几的HD或UNC 路徑
sqlcmd -S 10.1.2.3 -U sa -P 'password' -Q "BACKUP DATABASE dbname TO DISK = '\\%tmp%\Report.bak' WITH NOFORMAT, NOINIT, NAME = 'dbname-full', SKIP, NOREWIND, NOUNLOAD, STATS = 10"
https://learn.microsoft.com/zh-tw/sql/linux/sql-server-linux-setup-tools?view=sql-server-ver16#RHEL
訂閱：
文章 (Atom)