kk的blog: curl

顯示具有 curl 標籤的文章。顯示所有文章

2025/06/22

今天把oracle linux 8 升到 almalinux 9

首先升到 almalinux 8

dnf update -y

curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh

bash almalinux-deploy.sh

再升到 almalinux 9

dnf -y update

yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm

dnf install -y leapp-upgrade leapp-data-almalinux

leapp preupgrade

查看 log 解決升級會碰到的問題

cpu要改成 x86-64-v2 或以上

leapp upgrade

升完後

dnf -y update 出現以下訊息

warning: Signature not supported. Hash algorithm SHA1 not available.

需要清除有問題的 gpg-pubkey

列出所有 gpg key

rpm -qa gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'

刪除以下二個 gpg key

gpg-pubkey-ad986da3-5cabf60d gpg(Oracle OSS group (Open Source Software group) <build@oss.oracle.com>)

gpg-pubkey-ced7258b-6525146f gpg(AlmaLinux OS 8 <packager@almalinux.org>)

rpm -e gpg-pubkey-ad986da3-5cabf60d

rpm -e gpg-pubkey-ced7258b-6525146f

目前正常

再觀察看看

https://wiki.almalinux.org/documentation/migration-guide.html#migrating-using-almalinux-public-repositories-online-systems

https://www.veeble.com/kb/how-to-upgrade-almalinux-8-to-almalinux-9/

2024/11/29

使用curl 在 telegram傳送包含html tag 訊息的範例如下

不過支援的tag不多主要是要加上 &parse_mode=HTML

curl -X POST "https://api.telegram.org/bot711025:AAFcn2dfjsldfkjsdlfjdkfcvZo/sendMessage" -d "chat_id=-100236747688384&text=<u>test</u>&parse_mode=HTML"

2024/11/27

n8n裡有二個重要的東西一定要備分

一個是 credentials 一個是 workflow

指令如下

首先進到docker的 os

docker exec -it n8n /bin/sh

備分 credentials

n8n export:credentials --all --decrypt --output=exported-credentials.json

記得一定要用 --decrypt

如果不加的話因為每一台的 crypt key不同匯入別台新的几器會有問題

備分 workflow

n8n export:workflow --all --output=exported-workflows.json

也可以使用 api 拿出來

記得要先產生api key

語法如下

curl -X 'GET' \

'http://10.0.0.1:5678/api/v1/workflows?active=true&tags=test,production&name=My%20Workflow&projectId=your_api_key' \

-H 'accept: application/json'

備分檔可以傳出來或建個 workflow 定時丟出來

2024/11/06

graylog 6版後的index rotation改成了另一種方式

原本是設定

Max. days in storage

180

Min. days in storage

主觀認知是能查到60天內的資料超過60天系統會封存但應該還是能查到

但事實上超過60天就查不到資料了

為了符合法規規定所以現在改成

Max. days in storage

181

Min. days in storage

180

覺得舊版的方式比較好

能自己決定几天後就把 index close

然後再看狀況手動去砍

認真查了一下才發現graylog現在分成了三個版本

open的這個版本現在大幅減少支援的功能

為了解決以上的問題

看來只能定期手動或用crontab去close index

shell 如下

for i in {171..180}

echo $i

curl -u use:passwd -XPOST "http://10.0.0.1:9200/graylog_$i/_close"

done

目前觀察大約是每二天會 index rotate 一次

需要查詢舊資料時再打開

curl -v -X POST -H "Content-Type: application/json" -H "X-Requested-By: XMLHttpRequest" -u user:passwd http://10.0.0.1:9000/api/system/indexer/indices/graylog_245/reopen

https://graylog.org/post/understanding-data-tiering-in-60-seconds/

https://graylog.org/pricing/

2024/05/08

使用 curl 取得ip的相關資訊

curl ipinfo.io/8.8.8.8

{

"ip": "8.8.8.8",

"hostname": "dns.google",

"anycast": true,

"city": "Mountain View",

"region": "California",

"country": "US",

"loc": "37.4056,-122.0775",

"org": "AS15169 Google LLC",

"postal": "94043",

"timezone": "America/Los_Angeles",

"readme": "https://ipinfo.io/missingauth"

}

2024/05/06

發生好几次電腦大量封包的傳輸

netflow的值都在几萬甚至几十萬

以往的做法都是先把ip block掉

等user自己來反應

但醬沒辦法在事發的當時撈到相關資料

到底是那個程式造成的

2023/10/14

最近nas因為更新發生nfs不能使用的問題

因此暫時把graylog搬到其他台還沒更新的nas上

搬完後發現ES變成red

下指令看一下是那些shards

curl -XGET localhost:9200/_cat/shards|grep UNASSIGNED

index.action 0 r UNASSIGNED

index.do 0 r UNASSIGNED

index.aspx 0 r UNASSIGNED

graylog_159 2 p UNASSIGNED

index.htm 0 r UNASSIGNED

index.py 0 r UNASSIGNED

index.php 0 r UNASSIGNED

index.cgi 0 r UNASSIGNED

index.html 0 r UNASSIGNED

index.cfm 0 r UNASSIGNED

index.pl 0 r UNASSIGNED

index.jsp 0 r UNASSIGNED

index.asp 0 r UNASSIGNED

graylog_159 這個是放資料的直接砍了就損失一天的log

curl -XDELETE 'localhost:9200/graylog_159/'

此時ES已經變 yellow

但其他的shards也不知道砍了會不會有問題

forum上說的是因為沒有第二台可以replication所以會出現 UNASSIGNED

如果覺得礙眼不想看到可以取消 replication

指令如下

curl -X PUT "http://localhost:9200/index_name/_settings" -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":0}}'

目前就先醬放著吧

再觀察看看

https://community.graylog.org/t/graylog-opensearch-cluster-is-yellow/29678/4

2023/04/09

在 proxmox backup server 中使用 api 撈取相關資料

先建立 API token 要記住 token

再設定權限

使用 curl 撈取相關資料範例如下

curl --location --insecure --request GET 'https://10.0.0.1:8007/api2/json/nodes/{localhost}/tasks' --header 'Authorization: PBSAPIToken=root@pam!abc:your_token'|jq

API相關資料路徑可參考以下原廠連結

https://pbs.proxmox.com/docs/api-viewer/index.html

2023/03/02

使用curl 撈取 loki 資料的語法

依需求需要更改之處

job="abc"

查詢的關鍵字 192.168.1.2

查詢的區間

curl -G -s "http://10.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="abc"} |~ "192.168.1.2"' --data-urlencode "start=$(date -u +'%Y-%m-%dT%H:%M:%SZ' -d '-8 hour')" --data-urlencode "end=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"

curl -G -s "http://10.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="abc"} |~ "192.168.1.2"' --data-urlencode "start=$(date -u +'%Y-%m-%dT%H:%M:%SZ' -d '-7 day')" --data-urlencode "end=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"|jq

2022/09/10

今天想使用 line 發群組通知

首先申請權杖

官方文件範例是使用 curl

$ curl -X POST -H 'Authorization: Bearer <access_token>' -F 'message=foobar' \

https://notify-api.line.me/api/notify

但試了半天發現一個問題就是 message 無法換行

找了好久才找到範例如下

第一個方法

curl -X POST -H 'Authorization: Bearer (your token)' --data-binary message="%0A中文%0A123%0Aabc%0A456" https://notify-api.line.me/api/notify

第二個方法

curl -X POST -H 'Authorization: Bearer (your token)' -d $'message=123\nabc' https://notify-api.line.me/api/notify

使用python範例如下

檔名存為 line_notify.py

檔案內容如下

import requests

import sys

f = open(sys.argv[1])

msg = f.read()

f.close

def lineNotify(token, msg):

url = "https://notify-api.line.me/api/notify"

headers = {

"Authorization": "Bearer " + token

}

payload = {'message': msg}

r = requests.post(url, headers=headers, data=payload)

return r.status_code

token = "your token"

lineNotify(token, msg)

只要指定檔案發送

檔案內是什麼內容

發送就是什麼內容

用法

python line_notify.py /tmp/txt_to_send

https://xenby.com/b/274-%E6%95%99%E5%AD%B8-%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8-line-notify-%E5%B0%8D%E7%BE%A4%E7%B5%84%E9%80%B2%E8%A1%8C%E9%80%9A%E7%9F%A5

https://officeguide.cc/python-line-notify-send-messages-images-tutorial-examples/

https://fm-aid.com/bbs2/viewtopic.php?pid=52815

https://notify-bot.line.me/zh_TW/

https://notify-bot.line.me/doc/en/

2022/04/18

昨天是舊憑証的到期日

所以今天要撈出那些ip已經上了新憑証

把封包導入解密

使用 nmap 查找憑証指令如下

再找出使用 TWCA 的憑証並列出憑証到期日

#!/bin/bash

for i in {1..250}

echo $i

nmap -Pn --script ssl-cert -p 443 10.0.0.$i -oN ca_$i

done

若把執行結果餵到變數注意以下不同

r=`nmap --script ssl-cert -p 443 10.0.0.$i|grep -E 'TWCA|after'`

echo $r > ca_$i #導出結果無換行

echo "$r" > ca_$i #導出結果有換行

使用 curl 程式如下

#!/bin/bash

for i in {1..250}

echo $i

curl -m 3 https://10.0.0.$i -k -v -s -o /dev/null 2> /tmp/ca/ca_$i

done

接下來再針對關鍵字及有無過期查找即可

2021/07/14

今天利用API試著從 twse 撈出資料

curl https://mis.twse.com.tw/stock/api/getStockInfo.jsp?ex_ch=tse_0056.tw|jq

{

"msgArray": [

{

"tv": "-",

"ps": "-",

"nu": "http://www.yuantaetfs.com/#/RtNav/Index",

"pz": "-",

"bp": "0",

"a": "34.3800_34.3900_34.4000_34.4100_34.4200_",

"b": "34.3700_34.3600_34.3500_34.3400_34.3300_",

"c": "0056",

"d": "20210714",

"ch": "0056.tw",

"tlong": "1626229079000",

"f": "446_1402_1414_406_48_",

"ip": "0",

"g": "70_209_677_95_78_",

"mt": "848857",

"h": "34.9000",

"it": "02",

"l": "34.3500",

"n": "元大高股息",

"o": "34.8900",

"p": "0",

"ex": "tse",

"s": "-",

"t": "10:17:59",

"u": "38.3300",

"v": "14561",

"w": "31.3700",

"nf": "元大臺灣高股息證券投資信託基金",

"y": "34.8500",

"z": "-",

"ts": "0"

}

"referer": "",

"userDelay": 5000,

"rtcode": "0000",

"queryTime": {

"sysDate": "20210714",

"stockInfoItem": 901,

"stockInfo": 188897,

"sessionStr": "UserSession",

"sysTime": "10:18:03",

"showChart": false,

"sessionFromTime": -1,

"sessionLatestTime": -1

"rtmessage": "OK",

"exKey": "if_tse_0056.tw_zh-tw.null",

"cachedAlive": 82099

}

如果要取得msgArray裡的資料

指令如下

curl https://mis.twse.com.tw/stock/api/getStockInfo.jsp?ex_ch=tse_00882.tw|jq '.msgArray'

[

{

"tv": "-",

"ps": "-",

"nu": "https://www.ctbcinvestments.com/Product/ETFBusiness",

"pz": "-",

"bp": "0",

"a": "15.4100_15.4200_15.4300_15.4400_15.4500_",

"b": "15.4000_15.3900_15.3800_15.3700_15.3600_",

"c": "00882",

"d": "20210714",

"ch": "00882.tw",

"tlong": "1626229529000",

"f": "1005_1039_946_454_514_",

"ip": "0",

"g": "3086_873_1726_575_596_",

"mt": "353051",

"h": "15.4300",

"it": "02",

"l": "15.4000",

"n": "中信中國高股息",

"o": "15.4300",

"p": "0",

"ex": "tse",

"s": "-",

"t": "10:25:29",

"u": "9999.9500",

"v": "17761",

"nf": "中國信託全球收益ETF傘型證券投資信託基金之中國信託恒生中國高股息ETF證券投資信託基金",

"y": "15.4500",

"z": "-",

"ts": "0"

}

]

再來要報得目前的即時報價在 "a" 這個欄位這裡花了一點時間試

因為上一步的資料裡多了中括號 [ ]

所以指令要改成如下

curl https://mis.twse.com.tw/stock/api/getStockInfo.jsp?ex_ch=tse_0056.tw|jq '.msgArray[].a'

"34.3700_34.3800_34.3900_34.4000_34.4100_"

第一個分隔就是目前報價

https://zys-notes.blogspot.com/2020/01/api.html

2021/06/06

安裝jitsi的流程記錄一下 ubuntu 20.04

os安裝好後

apt update

apt upgrade -y

在DNS上設定好server的name

接下來

apt install curl gnupg

curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg --dearmor > /usr/share/keyrings/jitsi-keyring.gpg'

echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable/' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null

sudo apt-get -y update

sudo apt-get -y install jitsi-meet

如果要使用letsencrypt

apt install certbot

/usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh

重啟nginx

到 https://servername

就可以使用了

letsencrypt要記得每三個月renew一次

或使用crontab

1 1 * * 6 /usr/bin/certbot renew

https://kafeiou.pw/2020/06/19/2489/

https://campus-xoops.tn.edu.tw/modules/tad_book3/page.php?tbdsn=1557

2021/03/19

以往如果需要解壓後執行批次檔時

都是先壓成7z再利用7zsfx二次處理

今天發現 bandizip 可以直接壓成自解解檔達成以上的需求

但有個問題就是自解檔執行後會跳出詢問視窗而且預設無法關掉

找了一下forum

在執行時加上 /auto就可以解決這個問題

例如

abc.exe /auto

2024/3/3 後記

可以用批次檔處理這個問題

run.bat 內容如下

curl -o %tmp%\abc.exe http://10.0.0.1/abc.exe

start %tmp%\abc.exe /auto

https://www.azofreeware.com/2012/07/7-zip-sfx-maker-32-7z.html

Bandizip

https://groups.google.com/g/bandizip-win/c/tS9KLKh45O8/m/spdej7MYAwAJ

2020/12/14

之前升級graylog rest 碰到的問題

http://adminkk.blogspot.com/2020/11/graylog-4-ova-ubuntu-18.html

官方文件上說明此種方法會停止支援

必須使用新方法

但官方文件上並沒有很詳細的說明

去forum上問了

感謝回答

語法如下

直接匯出txt

絕對時間的語法

curl -u admin:passwd -H 'Accept: text/csv' -H "Content-Type:application/json" -H "Accept:application/json" -H 'X-Requested-By: cli' -d '{"streams":["000000000000000000000001"],"timerange":["absolute",{"from":"2020-12-11T00:00:00.000Z","to":"2020-12-11T01:00:00.000Z"}],"query_string":{"type":"elasticsearch","query_string":"keyword" }}' "http://10.0.0.1:9000/api/views/search/messages"

相對時間的語法

curl -u admin:passwd -H 'Accept: text/csv' -H "Content-Type:application/json" -H "Accept:application/json" -H 'X-Requested-By: cli' -d '{"streams":["000000000000000000000001"],"timerange":{"type": "relative","range": 300},"query_string":{"type":"elasticsearch","query_string":"keyword" }}' "http://10.0.0.1:9000/api/views/search/messages"

https://community.graylog.org/t/how-to-search-messages-using-rest-api/17943

2020/10/03

virustotal 有提供rest api 查詢

免費註冊後就可以在一定限制下免費使用

v2 語法如下

curl --request GET --url 'https://www.virustotal.com/vtapi/v2/ip-address/report?apikey="your api key"&ip=59.177.37.217'|jq

查詢結果如下

{

"asn": 17813,

"undetected_urls": [],

"undetected_downloaded_samples": [],

"country": "IN",

"response_code": 1,

"as_owner": "Mahanagar Telephone Nigam Limited",

"verbose_msg": "IP address in dataset",

"detected_downloaded_samples": [

{

"date": "2020-09-20 23:45:20",

"positives": 21,

"total": 72,

"sha256": "b5cf68c7cb5bb2d21d60bf6654926f61566d95bfd7c9f9e182d032f1da5b4605"

{

"date": "2020-09-20 20:37:59",

"positives": 33,

"total": 72,

"sha256": "c672798dca67f796972b42ad0c89e25d589d2e70eb41892d26adbb6a79f63887"

}

"detected_urls": [

{

"url": "http://59.177.37.217/",

"positives": 7,

"total": 79,

"scan_date": "2020-10-02 18:05:53"

{

"url": "http://59.177.37.217:58256/Mozi.m",

"positives": 8,

"total": 79,

"scan_date": "2020-09-24 02:04:33"

{

"url": "http://59.177.37.217:41901/Mozi.a",

"positives": 8,

"total": 79,

"scan_date": "2020-09-22 16:49:18"

{

"url": "https://59.177.37.217/",

"positives": 4,

"total": 79,

"scan_date": "2020-09-22 13:34:26"

{

"url": "http://59.177.37.217:39302/Mozi.m",

"positives": 9,

"total": 79,

"scan_date": "2020-09-20 21:52:26"

{

"url": "http://59.177.37.217/mozi.m",

"positives": 2,

"total": 79,

"scan_date": "2020-09-16 17:56:36"

{

"url": "http://59.177.37.217:58256/Mozi.m/",

"positives": 1,

"total": 79,

"scan_date": "2020-09-16 14:35:11"

}

"resolutions": []

}

v3 語法如下

curl --request GET --url 'https://www.virustotal.com/api/v3/search?query=209.59.217.36' --header 'x-apikey: your api key'

{

"data": [

{

"attributes": {

"as_owner": "The Endurance International Group, Inc.",

"asn": 29873,

"continent": "NA",

"country": "US",

"last_analysis_results": {

"ADMINUSLabs": {

"category": "harmless",

"engine_name": "ADMINUSLabs",

"method": "blacklist",

"result": "clean"

"AegisLab WebGuard": {

"category": "harmless",

"engine_name": "AegisLab WebGuard",

"method": "blacklist",

"result": "clean"

"AlienVault": {

"category": "harmless",

"engine_name": "AlienVault",

"method": "blacklist",

"result": "clean"

"Antiy-AVL": {

"category": "harmless",

"engine_name": "Antiy-AVL",

"method": "blacklist",

"result": "clean"

"AutoShun": {

"category": "harmless",

"engine_name": "AutoShun",

"method": "blacklist",

"result": "clean"

"Avira": {

"category": "harmless",

"engine_name": "Avira",

"method": "blacklist",

"result": "clean"

"BADWARE.INFO": {

"category": "harmless",

"engine_name": "BADWARE.INFO",

"method": "blacklist",

"result": "clean"

"Baidu-International": {

"category": "harmless",

"engine_name": "Baidu-International",

"method": "blacklist",

"result": "clean"

"BitDefender": {

"category": "harmless",

"engine_name": "BitDefender",

"method": "blacklist",

"result": "clean"

"Blueliv": {

"category": "harmless",

"engine_name": "Blueliv",

"method": "blacklist",

"result": "clean"

"CINS Army": {

"category": "harmless",

"engine_name": "CINS Army",

"method": "blacklist",

"result": "clean"

"CLEAN MX": {

"category": "harmless",

"engine_name": "CLEAN MX",

"method": "blacklist",

"result": "clean"

"CRDF": {

"category": "harmless",

"engine_name": "CRDF",

"method": "blacklist",

"result": "clean"

"Certego": {

"category": "harmless",

"engine_name": "Certego",

"method": "blacklist",

"result": "clean"

"Comodo Valkyrie Verdict": {

"category": "harmless",

"engine_name": "Comodo Valkyrie Verdict",

"method": "blacklist",

"result": "clean"

"CyRadar": {

"category": "harmless",

"engine_name": "CyRadar",

"method": "blacklist",

"result": "clean"

"Cyan": {

"category": "harmless",

"engine_name": "Cyan",

"method": "blacklist",

"result": "clean"

"CyberCrime": {

"category": "harmless",

"engine_name": "CyberCrime",

"method": "blacklist",

"result": "clean"

"DNS8": {

"category": "harmless",

"engine_name": "DNS8",

"method": "blacklist",

"result": "clean"

"Dr.Web": {

"category": "harmless",

"engine_name": "Dr.Web",

"method": "blacklist",

"result": "clean"

"ESET": {

"category": "harmless",

"engine_name": "ESET",

"method": "blacklist",

"result": "clean"

"ESTsecurity-Threat Inside": {

"category": "harmless",

"engine_name": "ESTsecurity-Threat Inside",

"method": "blacklist",

"result": "clean"

"EmergingThreats": {

"category": "harmless",

"engine_name": "EmergingThreats",

"method": "blacklist",

"result": "clean"

"Emsisoft": {

"category": "harmless",

"engine_name": "Emsisoft",

"method": "blacklist",

"result": "clean"

"EonScope": {

"category": "harmless",

"engine_name": "EonScope",

"method": "blacklist",

"result": "clean"

"Forcepoint ThreatSeeker": {

"category": "harmless",

"engine_name": "Forcepoint ThreatSeeker",

"method": "blacklist",

"result": "clean"

"Fortinet": {

"category": "harmless",

"engine_name": "Fortinet",

"method": "blacklist",

"result": "clean"

"FraudScore": {

"category": "harmless",

"engine_name": "FraudScore",

"method": "blacklist",

"result": "clean"

"G-Data": {

"category": "harmless",

"engine_name": "G-Data",

"method": "blacklist",

"result": "clean"

"Google Safebrowsing": {

"category": "harmless",

"engine_name": "Google Safebrowsing",

"method": "blacklist",

"result": "clean"

"GreenSnow": {

"category": "harmless",

"engine_name": "GreenSnow",

"method": "blacklist",

"result": "clean"

"Hoplite Industries": {

"category": "harmless",

"engine_name": "Hoplite Industries",

"method": "blacklist",

"result": "clean"

"IPsum": {

"category": "harmless",

"engine_name": "IPsum",

"method": "blacklist",

"result": "clean"

"K7AntiVirus": {

"category": "harmless",

"engine_name": "K7AntiVirus",

"method": "blacklist",

"result": "clean"

"Kaspersky": {

"category": "harmless",

"engine_name": "Kaspersky",

"method": "blacklist",

"result": "clean"

"Lumu": {

"category": "harmless",

"engine_name": "Lumu",

"method": "blacklist",

"result": "clean"

"MalSilo": {

"category": "harmless",

"engine_name": "MalSilo",

"method": "blacklist",

"result": "clean"

"Malware Domain Blocklist": {

"category": "harmless",

"engine_name": "Malware Domain Blocklist",

"method": "blacklist",

"result": "clean"

"MalwareDomainList": {

"category": "harmless",

"engine_name": "MalwareDomainList",

"method": "blacklist",

"result": "clean"

"MalwarePatrol": {

"category": "harmless",

"engine_name": "MalwarePatrol",

"method": "blacklist",

"result": "clean"

"Malwared": {

"category": "harmless",

"engine_name": "Malwared",

"method": "blacklist",

"result": "clean"

"Netcraft": {

"category": "harmless",

"engine_name": "Netcraft",

"method": "blacklist",

"result": "clean"

"NotMining": {

"category": "harmless",

"engine_name": "NotMining",

"method": "blacklist",

"result": "clean"

"Nucleon": {

"category": "harmless",

"engine_name": "Nucleon",

"method": "blacklist",

"result": "clean"

"OpenPhish": {

"category": "harmless",

"engine_name": "OpenPhish",

"method": "blacklist",

"result": "clean"

"PREBYTES": {

"category": "harmless",

"engine_name": "PREBYTES",

"method": "blacklist",

"result": "clean"

"PhishLabs": {

"category": "harmless",

"engine_name": "PhishLabs",

"method": "blacklist",

"result": "clean"

"Phishing Database": {

"category": "harmless",

"engine_name": "Phishing Database",

"method": "blacklist",

"result": "clean"

"Phishtank": {

"category": "harmless",

"engine_name": "Phishtank",

"method": "blacklist",

"result": "clean"

"Quick Heal": {

"category": "harmless",

"engine_name": "Quick Heal",

"method": "blacklist",

"result": "clean"

"Quttera": {

"category": "harmless",

"engine_name": "Quttera",

"method": "blacklist",

"result": "clean"

"SCUMWARE.org": {

"category": "harmless",

"engine_name": "SCUMWARE.org",

"method": "blacklist",

"result": "clean"

"SecureBrain": {

"category": "harmless",

"engine_name": "SecureBrain",

"method": "blacklist",

"result": "clean"

"Segasec": {

"category": "harmless",

"engine_name": "Segasec",

"method": "blacklist",

"result": "clean"

"Snort IP sample list": {

"category": "suspicious",

"engine_name": "Snort IP sample list",

"method": "blacklist",

"result": "suspicious"

"Sophos": {

"category": "harmless",

"engine_name": "Sophos",

"method": "blacklist",

"result": "clean"

"Spam404": {

"category": "harmless",

"engine_name": "Spam404",

"method": "blacklist",

"result": "clean"

"Spamhaus": {

"category": "harmless",

"engine_name": "Spamhaus",

"method": "blacklist",

"result": "clean"

"StopBadware": {

"category": "harmless",

"engine_name": "StopBadware",

"method": "blacklist",

"result": "clean"

"StopForumSpam": {

"category": "harmless",

"engine_name": "StopForumSpam",

"method": "blacklist",

"result": "clean"

"Sucuri SiteCheck": {

"category": "harmless",

"engine_name": "Sucuri SiteCheck",

"method": "blacklist",

"result": "clean"

"Tencent": {

"category": "harmless",

"engine_name": "Tencent",

"method": "blacklist",

"result": "clean"

"ThreatHive": {

"category": "harmless",

"engine_name": "ThreatHive",

"method": "blacklist",

"result": "clean"

"Threatsourcing": {

"category": "suspicious",

"engine_name": "Threatsourcing",

"method": "blacklist",

"result": "suspicious"

"Trustwave": {

"category": "harmless",

"engine_name": "Trustwave",

"method": "blacklist",

"result": "clean"

"URLhaus": {

"category": "harmless",

"engine_name": "URLhaus",

"method": "blacklist",

"result": "clean"

"VX Vault": {

"category": "harmless",

"engine_name": "VX Vault",

"method": "blacklist",

"result": "clean"

"Virusdie External Site Scan": {

"category": "harmless",

"engine_name": "Virusdie External Site Scan",

"method": "blacklist",

"result": "clean"

"Web Security Guard": {

"category": "harmless",

"engine_name": "Web Security Guard",

"method": "blacklist",

"result": "clean"

"Yandex Safebrowsing": {

"category": "harmless",

"engine_name": "Yandex Safebrowsing",

"method": "blacklist",

"result": "clean"

"ZeroCERT": {

"category": "harmless",

"engine_name": "ZeroCERT",

"method": "blacklist",

"result": "clean"

"desenmascara.me": {

"category": "harmless",

"engine_name": "desenmascara.me",

"method": "blacklist",

"result": "clean"

"malwares.com URL checker": {

"category": "harmless",

"engine_name": "malwares.com URL checker",

"method": "blacklist",

"result": "clean"

"securolytics": {

"category": "harmless",

"engine_name": "securolytics",

"method": "blacklist",

"result": "clean"

"zvelo": {

"category": "harmless",

"engine_name": "zvelo",

"method": "blacklist",

"result": "clean"

}

"last_analysis_stats": {

"harmless": 74,

"malicious": 0,

"suspicious": 2,

"timeout": 0,

"undetected": 0

"last_modification_date": 1601705254,

"network": "209.59.192.0/19",

"regional_internet_registry": "ARIN",

"reputation": 0,

"tags": [],

"total_votes": {

"harmless": 0,

"malicious": 0

"whois": "NetRange: 209.59.192.0 - 209.59.223.255\nCIDR: 209.59.192.0/19\nNetName: BIZLAND-FC02\nNetHandle: NET-209-59-192-0-1\nParent: NET209 (NET-209-0-0-0-0)\nNetType: Direct Allocation\nOriginAS: AS29873\nOrganization: The Endurance International Group, Inc. (EIG-12)\nRegDate: 2004-07-30\nUpdated: 2012-03-02\nRef: https://rdap.arin.net/registry/ip/209.59.192.0\nOrgName: The Endurance International Group, Inc.\nOrgId: EIG-12\nAddress: 10 Corporate Drive\nAddress: Suite 300\nCity: Burlington\nStateProv: MA\nPostalCode: 01803\nCountry: US\nRegDate: 2005-02-07\nUpdated: 2018-06-14\nRef: https://rdap.arin.net/registry/entity/EIG-12\nOrgTechHandle: EIGAR-ARIN\nOrgTechName: eig-arin\nOrgTechPhone: +1-866-897-5421 \nOrgTechEmail: eig-arin@endurance.com\nOrgTechRef: https://rdap.arin.net/registry/entity/EIGAR-ARIN\nOrgAbuseHandle: EIGAB-ARIN\nOrgAbuseName: eig-abuse\nOrgAbusePhone: +1-877-659-6181 \nOrgAbuseEmail: eig-abuse@endurance.com\nOrgAbuseRef: https://rdap.arin.net/registry/entity/EIGAB-ARIN\nOrgNOCHandle: ENO91-ARIN\nOrgNOCName: EIG Network Operations\nOrgNOCPhone: +1-877-659-6181 \nOrgNOCEmail: eig-noc@endurance.com\nOrgNOCRef: https://rdap.arin.net/registry/entity/ENO91-ARIN\n",

"whois_date": 1568088719

"id": "209.59.217.36",

"links": {

"self": "https://www.virustotal.com/api/v3/ip_addresses/209.59.217.36"

"type": "ip_address"

}

"links": {

"self": "https://www.virustotal.com/api/v3/search?query=209.59.217.36"

}

方便用來判斷該ip是否為惡意ip

https://developers.virustotal.com/reference

https://developers.virustotal.com/v3.0/reference

2019/11/01

今天模擬在一台graylog server 三個ES node的環境

然後當有一台ES fail時要怎麼處理

首先我們先看一下目前ES的狀況

有三個node

status也是green

curl -XGET http://192.168.12.201:9200/_cluster/health?pretty

{

"cluster_name" : "graylog",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 3,

"number_of_data_nodes" : 3,

"active_primary_shards" : 11,

"active_shards" : 14,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0,

"delayed_unassigned_shards" : 0,

"number_of_pending_tasks" : 0,

"number_of_in_flight_fetch" : 0,

"task_max_waiting_in_queue_millis" : 0,

"active_shards_percent_as_number" : 100.0

}

再來看一下目前所有shards的狀況

curl -XGET 192.168.12.203:9200/_cat/shards

gl-events_0 3 p STARTED 0 230b 192.168.12.202 es-node-02

gl-events_0 2 p STARTED 0 230b 192.168.12.201 es-node-1

gl-events_0 1 p STARTED 0 230b 192.168.12.203 es-node-03

gl-events_0 0 p STARTED 0 230b 192.168.12.202 es-node-02

graylog_3 2 r STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 2 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 1 r STARTED 1 7kb 192.168.12.202 es-node-02

graylog_3 1 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 0 p STARTED 1 7kb 192.168.12.202 es-node-02

graylog_3 0 r STARTED 1 7kb 192.168.12.203 es-node-03

gl-system-events_0 3 p STARTED 0 230b 192.168.12.203 es-node-03

gl-system-events_0 2 p STARTED 0 230b 192.168.12.202 es-node-02

gl-system-events_0 1 p STARTED 0 230b 192.168.12.201 es-node-1

gl-system-events_0 0 p STARTED 0 230b 192.168.12.203 es-node-03

我們關掉其中一個ES node 192.168.12.202 模擬故障

查看整個cluster狀況

nodes變成2

status也變為red

curl -XGET http://192.168.12.201:9200/_cluster/health?pretty

{

"cluster_name" : "graylog",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 2,

"number_of_data_nodes" : 2,

"active_primary_shards" : 8,

"active_shards" : 9,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 5,

"delayed_unassigned_shards" : 5,

"number_of_pending_tasks" : 0,

"number_of_in_flight_fetch" : 0,

"task_max_waiting_in_queue_millis" : 0,

"active_shards_percent_as_number" : 64.28571428571429

}

再來看一下shards的狀況

192.168.12.202這個node己經不見了

而且上面的shards 變成 UNASSIGNED

curl -XGET 192.168.12.203:9200/_cat/shards

gl-system-events_0 3 p STARTED 0 261b 192.168.12.203 es-node-03

gl-system-events_0 2 p UNASSIGNED

gl-system-events_0 1 p STARTED 0 261b 192.168.12.201 es-node-1

gl-system-events_0 0 p STARTED 0 261b 192.168.12.203 es-node-03

graylog_3 2 r STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 2 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 1 r STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 1 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 0 p STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 0 r STARTED 1 7kb 192.168.12.201 es-node-1

gl-events_0 3 p UNASSIGNED

gl-events_0 2 p STARTED 0 261b 192.168.12.201 es-node-1

gl-events_0 1 p STARTED 0 261b 192.168.12.203 es-node-03

gl-events_0 0 p UNASSIGNED

找一台机器重裝ES後並重新加入cluster

先看一下狀況

nodes己經回來變成3了

可是status還是red

curl -XGET http://192.168.12.201:9200/_cluster/health?pretty

{

"cluster_name" : "graylog",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 3,

"number_of_data_nodes" : 3,

"active_primary_shards" : 8,

"active_shards" : 11,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 3,

"delayed_unassigned_shards" : 0,

"number_of_pending_tasks" : 0,

"number_of_in_flight_fetch" : 0,

"task_max_waiting_in_queue_millis" : 0,

"active_shards_percent_as_number" : 78.57142857142857

}

再來看shards的狀況

還是 UNASSIGNED 並沒有復原

curl -XGET 192.168.12.203:9200/_cat/shards

gl-system-events_0 3 p STARTED 0 261b 192.168.12.203 es-node-03

gl-system-events_0 2 p UNASSIGNED

gl-system-events_0 1 p STARTED 0 261b 192.168.12.201 es-node-1

gl-system-events_0 0 p STARTED 0 261b 192.168.12.203 es-node-03

graylog_3 2 r STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 2 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 1 r STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 1 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 0 p STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 0 r STARTED 1 7kb 192.168.12.201 es-node-1

gl-events_0 3 p UNASSIGNED

gl-events_0 2 p STARTED 0 261b 192.168.12.201 es-node-1

gl-events_0 1 p STARTED 0 261b 192.168.12.203 es-node-03

gl-events_0 0 p UNASSIGNED

查了資料說可以 reroute share

但實作上有問題無法執行

http://www.wklken.me/posts/2015/05/23/elasticsearch-issues.html

https://toutiao.io/posts/na8zgp/preview

目前試出來的做法是先關掉graylog server

systemctl stop graylog-server.service

接下來把所有的 UNASSIGNED 砍了

curl -XDELETE '192.168.12.201:9200/gl-system-events_0/'

curl -XDELETE '192.168.12.201:9200/gl-events_0'

砍完後再去看shards

curl -XGET 192.168.12.203:9200/_cat/shards

graylog_3 2 r STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 2 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 1 r STARTED 1 7kb 192.168.12.202 es-node-02

graylog_3 1 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 0 r STARTED 1 7kb 192.168.12.202 es-node-02

graylog_3 0 p STARTED 1 7kb 192.168.12.203 es-node-03

這個是原本的資料檔

而且有設定 Index replicas

重啟graylog server

graylog會把剛剛砍掉的 gl-system-events_0 gl-events_0 建回來

收集的資料是放在 graylog_* 所以不會有影響

再看一次shards

全部都正常了

curl -XGET 192.168.12.203:9200/_cat/shards

gl-system-events_0 3 p STARTED 0 230b 192.168.12.203 es-node-03

gl-system-events_0 2 p STARTED 0 230b 192.168.12.202 es-node-02

gl-system-events_0 1 p STARTED 0 230b 192.168.12.201 es-node-1

gl-system-events_0 0 p STARTED 0 230b 192.168.12.203 es-node-03

graylog_3 2 r STARTED 1 7kb 192.168.12.203 es-node-03

graylog_3 2 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 1 r STARTED 1 7kb 192.168.12.202 es-node-02

graylog_3 1 p STARTED 1 7kb 192.168.12.201 es-node-1

graylog_3 0 r STARTED 1 7kb 192.168.12.202 es-node-02

graylog_3 0 p STARTED 1 7kb 192.168.12.203 es-node-03

gl-events_0 3 p STARTED 0 230b 192.168.12.202 es-node-02

gl-events_0 2 p STARTED 0 230b 192.168.12.201 es-node-1

gl-events_0 1 p STARTED 0 230b 192.168.12.203 es-node-03

gl-events_0 0 p STARTED 0 230b 192.168.12.202 es-node-02

所以記得 Configure Index Set 要設定Index replicas 至少為1

Index shards的數量就根据你ES node的數量來設定

如果ES node 有三個就設定為3

2019/10/22

今天整理graylog的時候發現一件事

目前我在index set的設定是留180天
超過的delete
不過我會手動去close index
只留約一個月的資料查詢效率比較好
可是今天發現index超過了180天但系統並沒有去delete
我不確定是不是因為我手動去close的關係
如果真的是如此的話那政策就要改成定時close
然後再手動去delete了
手動砍的shell如下
每天找出最舊的index再砍掉

#!/bin/bash

del_idx=`/usr/bin/curl -u admin:password 'http://127.0.0.1:9200/_cat/indices/graylog*?v'|grep close|awk '{print $2}'|sort -rn -t_ -k 2|tail -n 1`

/usr/bin/curl -XDELETE "http://localhost:9200/$del_idx/"

用crontab 來做吧

2019/09/07

昨天早上接到有人問graylog的問題版本 3.0.2
說本來可以正常運作
但改完ip後
就收不到資料了
連進去看之後先df看了一下發現HD的使用率很高
然後ES的log出現以下的訊息

[INFO ][cluster.routing.allocation.decider] [Milan] low disk watermark [15%] exceeded on [DZqnmWIZRpapZY_TPkkMBw][Milan] free: 58.6gb[12.6%], replicas will not be assigned to this node

徴求user同意後先把一個indices砍了

curl -u admin:password -XDELETE http://10.1.2.3:9200/graylog_0

HD使用率就下降了一半

再來在管理介面上 rotate active write index

rotate active write index

可是下完後在管理介面上index就跑不出來了

去看了graylog的log出現以下訊息

elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, 'cluster_block_exception', 'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];')

只好再下指令 unlock

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

再關掉index

curl -u admin:password -XPOST 'http://10.1.2.3:9200/graylog_4/_close'

在管理介面上就可以看到index了

可是這個時候在search的介面出問題而不能search

因為下完之前的指令出現了二個open的index

以下是列出所有index的指令
curl -u admin:password 'http://10.1.2.3:9200/_cat/indices/graylog*?v'

所以關掉一個後就正常了

curl -u admin:password -XPOST 'http://10.1.2.3:9200/graylog_5/_close'

到此在log裡就沒有看到任何error

可是訊息還是沒看到而且process buffer一直在100%

用top去看 java也吃掉了大量的cpu

我在猜是不是extractor的問題

因為還要去處理別的問題

所以就先斷線了

後記

之後user有再連絡說log已經有進來了

可能是之前塞住的message消化完了

所以就再觀察看看

2017/12/21

最近在graylog上碰到搜尋都OK
除了要匯出成csv會很慢
不過這個問題應該跟接下來要討論的是同一個問題

使用rest撈資料時發生有時撈得到有時撈不到的情況
但確定指令沒下錯　而且確定語法一定有資料可以出來
於是寫了一個shell來試

#!/bin/bash
i=1
while :
do
echo $i
curl -m 240 -u user:pwd 'http://10.0.0.1:9000/api/search/universal/keyword?query=nf_dst_address%3A192.168.12.220&keyword=last%2020%20hours&fields=message&limit=1'
i=`expr $i + 1`
sleep 3
done

出現以下的結果

1
2
3
4
"timestamp","message"
"2017-12-20T10:09:23.000Z","NetFlowV5 [210.65.47.55]:443 <> [192.168.12.220]:42702 proto:6 pkts:1 bytes:52"
5
"timestamp","message"
"2017-12-20T10:09:23.000Z","NetFlowV5 [210.65.47.55]:443 <> [192.168.12.220]:42702 proto:6 pkts:1 bytes:52"
6
"timestamp","message"
"2017-12-20T10:09:23.000Z","NetFlowV5 [210.65.47.55]:443 <> [192.168.12.220]:42702 proto:6 pkts:1 bytes:52"
7
8
"timestamp","message"
"2017-12-20T10:09:23.000Z","NetFlowV5 [210.65.47.55]:443 <> [192.168.12.220]:42702 proto:6 pkts:1 bytes:52"
9
"timestamp","message"
"2017-12-20T10:09:23.000Z","NetFlowV5 [210.65.47.55]:443 <> [192.168.12.220]:42702 proto:6 pkts:1 bytes:52"
10
11
12
"timestamp","message"
"2017-12-20T10:09:23.000Z","NetFlowV5 [210.65.47.55]:443 <> [192.168.12.220]:42702 proto:6 pkts:1 bytes:52"
13
"timestamp","message"
"2017-12-20T10:09:23.000Z","NetFlowV5 [210.65.47.55]:443 <> [192.168.12.220]:42702 proto:6 pkts:1 bytes:52"

而且在管理介面上看到的狀況如下圖

找了很多資料　調了很多參數都沒用　直覺是個bug

訂閱：文章 (Atom)