Finology 大数据金融

通过大数据以量化金融

0%

发表于 更新于 分类于 阅读次数： 评论数：

Http请求百度

TCP连接

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# exec 5<> /dev/tcp/www.baidu.com/80
# echo -e 'GET / HTTP/1.0\n' >& 5
# cat <& 5

HTTP/1.0 200 OK
Accept-Ranges: bytes
Cache-Control: no-cache
Content-Length: 14615
Content-Type: text/html
Date: Sat, 21 Mar 2020 08:14:27 GMT
P3p: CP=" OTI DSP COR IVA OUR IND COM "
P3p: CP=" OTI DSP COR IVA OUR IND COM "
Pragma: no-cache
Server: BWS/1.1
Set-Cookie: BAIDUID=42C9E3F8B0C101BDEA241D5174987905:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BIDUPSID=42C9E3F8B0C101BDEA241D5174987905; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: PSTM=1584778467; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BAIDUID=42C9E3F8B0C101BDD0E9FB6623D5C445:FG=1; max-age=31536000; expires=Sun, 21-Mar-21 08:14:27 GMT; domain=.baidu.com; path=/; version=1; comment=bd
Traceid: 1584778467046850433011449469410765311951
Vary: Accept-Encoding
X-Ua-Compatible: IE=Edge,chrome=1

<!DOCTYPE html><!--STATUS OK-->
<html>
<head>
...

5 叫文件描述符

Socket 连接

1
2
3
4
5
6
7
8
9
# ss -antp
State       Recv-Q Send-Q                                                   Local Address:Port                                                                  Peer Address:Port
LISTEN      0      128                                                                  *:22                                                                               *:*                   users:(("sshd",pid=1007,fd=3))
LISTEN      0      100                                                          127.0.0.1:25                                                                               *:*                   users:(("master",pid=1166,fd=13))
ESTAB       0      0                                                        172.16.64.220:22                                                                     172.16.64.1:62779               users:(("sshd",pid=6702,fd=3))
ESTAB       0      0                                                        172.16.64.220:22                                                                     172.16.64.1:62925               users:(("sshd",pid=6820,fd=3))
CLOSE-WAIT  0      0                                                        172.16.64.220:57720                                                                 36.152.44.95:80                  users:(("ss",pid=6930,fd=5),("bash",pid=6706,fd=5))
LISTEN      0      128                                                                 :::22                                                                              :::*                   users:(("sshd",pid=1007,fd=4))
LISTEN      0      100                                                                ::1:25                                                                              :::*                   users:(("master",pid=1166,fd=14))

或者

1
2
3
4
5
6
7
8
9
10
# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1007/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1166/master
tcp        0      0 172.16.64.220:22        172.16.64.1:62779       ESTABLISHED 6702/sshd: root@pts
tcp        0      0 172.16.64.220:22        172.16.64.1:62925       ESTABLISHED 6820/sshd: root@pts
tcp        0      0 172.16.64.220:57720     36.152.44.95:80         CLOSE_WAIT  6706/-bash
tcp6       0      0 :::22                   :::*                    LISTEN      1007/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1166/master

查看内核一些参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3802
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3802
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

open files 1024 个

1
2
3
4
5
6
7
8
9
10
# cd /proc/$$/fd
# pwd
/proc/6706/fd
# ll
总用量 0
lrwx------. 1 root root 64 11月 22 18:54 0 -> /dev/pts/0
lrwx------. 1 root root 64 11月 22 18:54 1 -> /dev/pts/0
lrwx------. 1 root root 64 11月 22 18:54 2 -> /dev/pts/0
lrwx------. 1 root root 64 11月 22 19:01 255 -> /dev/pts/0
lrwx------. 1 root root 64 11月 22 18:54 5 -> socket:[49378]

修改

1
2
3
# ulimit -SHn 65535
# ulimit -n
65535

查看路由表

网络层 IP协议。携带的是目标地址的IP地址。

1
2
3
4
5
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.64.2     0.0.0.0         UG    100    0        0 ens33
172.16.64.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33

查看链路

如何找到下一跳地址，是通过 mac 地址来确定的。

1
2
3
4
5
# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.16.64.254            ether   00:50:56:e7:e8:58   C                     ens33
172.16.64.2              ether   00:50:56:eb:6f:44   C                     ens33
172.16.64.1              ether   00:50:56:c0:00:08   C                     ens33

请求百度抓包

再打开另一个终端

使用 tcpdump 命令监控

1
2
3
# tcpdump -nnn -i ens33 port 80 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes

如果没有找到命令就安装一下。

1
# yum install tcpdump

在之前的终端访问百度，在访问之前，我们先删除链路层的记录，让数据包被发送时找不到网关下一跳的 mac 地址。

1
# arp -d 172.16.64.2

删除以后，如果有访问网络，这个 arp 的记录很快又会生成。所以我们在监控时，要在删除记录后的瞬间请求百度。

1
# arp -d 172.16.64.2 && curl www.baidu.com

通过 tcpdump 我们可以看到抓包后的内容。172.16.64.220 是本机。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# tcpdump -nnn -i ens33 port 80 or arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
21:54:21.286505 ARP, Request who-has 172.16.64.2 tell 172.16.64.220, length 28
21:54:21.286832 ARP, Reply 172.16.64.2 is-at 00:50:56:eb:6f:44, length 46
21:54:21.316879 IP 172.16.64.220.57734 > 36.152.44.95.80: Flags [S], seq 2491677366, win 29200, options [mss 1460,sackOK,TS val 81477028 ecr 0,nop,wscale 7], length 0
21:54:21.330823 IP 36.152.44.95.80 > 172.16.64.220.57734: Flags [S.], seq 1247240682, ack 2491677367, win 64240, options [mss 1460], length 0
21:54:21.330911 IP 172.16.64.220.57734 > 36.152.44.95.80: Flags [.], ack 1, win 29200, length 0
21:54:21.331466 IP 172.16.64.220.57734 > 36.152.44.95.80: Flags [P.], seq 1:78, ack 1, win 29200, length 77: HTTP: GET / HTTP/1.1
21:54:21.331749 IP 36.152.44.95.80 > 172.16.64.220.57734: Flags [.], ack 78, win 64240, length 0
21:54:21.347623 IP 36.152.44.95.80 > 172.16.64.220.57734: Flags [P.], seq 1:2782, ack 78, win 64240, length 2781: HTTP: HTTP/1.1 200 OK
21:54:21.347688 IP 172.16.64.220.57734 > 36.152.44.95.80: Flags [.], ack 2782, win 33580, length 0
21:54:21.348071 IP 172.16.64.220.57734 > 36.152.44.95.80: Flags [F.], seq 78, ack 2782, win 33580, length 0
21:54:21.348981 IP 36.152.44.95.80 > 172.16.64.220.57734: Flags [.], ack 79, win 64239, length 0
21:54:21.362444 IP 36.152.44.95.80 > 172.16.64.220.57734: Flags [FP.], seq 2782, ack 79, win 64239, length 0
21:54:21.362479 IP 172.16.64.220.57734 > 36.152.44.95.80: Flags [.], ack 2783, win 33580, length 0
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel

mss 1460

发表于 更新于 分类于 ， 阅读次数： 评论数：

安装Redis

从Redis官网下载最新版本的redis。

1
$ sudo wget http://download.redis.io/releases/redis-5.0.8.tar.gz

如需安装 Redis 4 版本，可参考 Linux(CentOS)环境安装Redis 4

新建redis安装目录

1
$ sudo mkdir /usr/local/redis

解压Redis

1
$ sudo tar -zxvf redis-5.0.8.tar.gz -C /usr/local/redis

编译Redis

先安装gcc

1
$ sudo yum install gcc-c++

编译

1
2
cd /usr/local/redis/redis-5.0.8/
sudo make

安装

把redis安装在目录/usr/local/redis/redis-5.0.8/中，如果不写 PREFIX 参数，即默认安装在/usr/local/bin下面

1
2
3
4
5
6
7
8
9
10
$ cd src/
$ sudo make install PREFIX=/usr/local/redis/redis-5.0.8/

Hint: It's a good idea to run 'make test' ;)

    INSTALL install
    INSTALL install
    INSTALL install
    INSTALL install
    INSTALL install
1
2
3
$ make test
You need tcl 8.5 or newer in order to run the Redis test
make: *** [test] Error 1

安装tcl

1
$ sudo yum install tcl
1
2
3
4
sudo make test
\o/ All tests passed without errors!

Cleanup: may take some time... OK

安装完成后，在目录 /usr/local/redis/redis-5.0.8 下面会出现一个 bin 目录

1
2
3
4
5
6
7
8
$ ll bin/
total 32772
-rwxr-xr-x. 1 root root 4366824 Mar 20 13:14 redis-benchmark
-rwxr-xr-x. 1 root root 8125024 Mar 20 13:14 redis-check-aof
-rwxr-xr-x. 1 root root 8125024 Mar 20 13:14 redis-check-rdb
-rwxr-xr-x. 1 root root 4807816 Mar 20 13:14 redis-cli
lrwxrwxrwx. 1 root root      12 Mar 20 13:14 redis-sentinel -> redis-server
-rwxr-xr-x. 1 root root 8125024 Mar 20 13:14 redis-server

启动Redis服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ bin/redis-server 
36959:C 20 Mar 2020 13:28:02.467 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
36959:C 20 Mar 2020 13:28:02.467 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=36959, just started
36959:C 20 Mar 2020 13:28:02.467 # Warning: no config file specified, using the default config. In order to specify a config file use bin/redis-server /path/to/redis.conf
36959:M 20 Mar 2020 13:28:02.467 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
36959:M 20 Mar 2020 13:28:02.467 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
36959:M 20 Mar 2020 13:28:02.467 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 5.0.8 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 36959
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

36959:M 20 Mar 2020 13:28:02.468 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
36959:M 20 Mar 2020 13:28:02.468 # Server initialized
36959:M 20 Mar 2020 13:28:02.468 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
36959:M 20 Mar 2020 13:28:02.468 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
36959:M 20 Mar 2020 13:28:02.468 * Ready to accept connections

通过客户端连接redis服务

1
2
3
4
5
6
$ bin/redis-cli 
127.0.0.1:6379> set name key
OK
127.0.0.1:6379> get name
"key"
127.0.0.1:6379>

如果通过非127.0.0.1连接，会报如下错误：

1
2
$ bin/redis-cli -h 192.168.229.130
192.168.229.130:6379> get name
1
(error) DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.

修改配置文件

1
$ sudo vi redis.conf

修改 bind 参数

1
bind 0.0.0.0

重启 redis 服务

通过 -h 指定redis服务的地址

1
2
bin/redis-cli -h 192.168.229.130
192.168.229.130:6379>

运行完成以后，如果需要关闭redis服务

1
2
127.0.0.1:6379> SHUTDOWN SAVE
not connected>

发表于 更新于 分类于 ， 阅读次数： 评论数：

在Win10环境，Chrome版本为80时，会报错：喔唷，崩溃啦。

在 mac 环境没有遇到这种情况。

解决方法：

找到 Chrome 的快捷键，在执行命令的后面，加入如下参数：

1
--disable-features=RendererCodeIntegrity

再次打开，就可以了。