用nginx-upsync-module同步consul服务到nginx upstream

用<Centos7配置LNMP nginx10+mariadb+php5.6>安装nginx后,

是没有nginx-upsync-module的.如此一来可能还不如直接编译安装,

但是这样安装好处在于,可以方便使用systemtl来管理nginx,不需要自己去添加服务,懒人必备.

然后再找同样版本的nginx,编译安装一遍,加进自己的模块.

  • 安装编译工具全家桶 yum groupinstall 'Development Tools'

  • 下载源码

1
2
3
4
wget http://nginx.org/download/nginx-1.12.1.tar.gz
tar -zxvf nginx-1.12.1.tar.gz
git clone https://github.com/onecer/nginx-upsync-module.git
#git clone https://github.com/weibocom/nginx-upsync-module.git
  • 安装编译依赖组件yum install -y zlib zlib-devel openssl openssl-devel pcre pcre-devel

  • 编译nginx

添加模块 --add-module=../nginx-upsync-module 其它模块有需要的自己加

1
2
3
cd nginx-1.12.1
./configure --with-http_stub_status_module --with-http_ssl_module --with-http_v2_module --with-http_gzip_static_module --with-http_gunzip_module --with-stream --with-stream_ssl_module --with-http_sub_module --with-http_flv_module --with-http_addition_module --with-http_realip_module --with-http_mp4_module --with-stream --with-stream_ssl_module --add-module=../nginx-upsync-module
make && make install

注意路径,我这里安装好后,路径是/usr/local/nginx/

添加软连接到常用路径ln -s /usr/local/nginx/sbin/nginx /usr/sbin/nginx

添加服务文件:
vim /usr/lib/systemd/system/nginx.service

服务描述文件里面 /var/run/nginx.pid 一定要和 nginx.conf 地址对应上.不然启动获取不到pid.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target
  • 添加nginx配置

运行nginx

systemctl daemon-reload && systemctrl enable/start nginx

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
    upstream illstrator-plus-web {
server 127.0.0.1:11111;
upsync 127.0.0.1:8500/v1/health/service/illstrator-plus-web upsync_timeout=6m upsync_interval=500ms upsync_type=consul_health strong_dependency=off;
upsync_dump_path /usr/local/nginx/conf/servers/servers_illstrator-plus-web.conf;
}

server {
listen 80;
server_name xxx.com;

location / {
proxy_pass http://illstrator-plus-web/;
proxy_read_timeout 300s;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

发现行不通,服务我是部署在docker swarm里面的.我获取的服务数据是这样的.[ServiceAddress]里面注册的是内网IP

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[
{
"ID": "ce711dca-f8fb-f1c3-4043-66a08040255e",
"Node": "node1",
"Address": "172.31.93.37",
"Datacenter": "dc1",
"TaggedAddresses": {
"lan": "172.31.93.37",
"wan": "172.31.93.37"
},
"NodeMeta": {
"consul-network-segment": ""
},
"ServiceKind": "",
"ServiceID": "chahuashi:illstrator-plus-web.1.8gjsduc64vor8runu9dw31m4k:8816",
"ServiceName": "illstrator-plus-web",
"ServiceTags": [],
"ServiceAddress": "10.0.0.15",
"ServiceMeta": {
"port": "8816"
},
"ServicePort": 8816,
"ServiceEnableTagOverride": false,
"ServiceProxyDestination": "",
"ServiceConnect": {
"Native": false,
"Proxy": null
},
"CreateIndex": 41480,
"ModifyIndex": 41480
}
]

现在有几条路走.

  1. Fork nginx-upsync-module 的源码 修改 ngx_http_upsync_consul_health_parse_json(void *data) 函数的处理,让它返回node的Address

  2. 摒弃全自动,手动添加节点信息到Consul KV

  3. 别用这种方式了,直接添加nginx节点.

  4. nginx装集群内

我选了第二种方法.主要是第一种的话,固然方便点,但是最终还是走routing mesh,性能没有提升.第四种麻烦,第三种增加服务会中断服务才能更新.

最后还是用了第一种,对源码做了些小修改,上面源码已经是改过的,没改的源码地址,是注释掉的那个。

修改nginx配置

1
2
3
4
5
6
7
upstream illstrator-plus-web {
server 127.0.0.1:11111;
upsync 127.0.0.1:8500/v1/health/service/illstrator-plus-web upsync_timeout=6m upsync_interval=500ms upsync_type=consul_health strong_dependency=off;
# upsync 127.0.0.1:8500/v1/kv/upstreams/illstrator-plus-web upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
upsync_dump_path /usr/local/nginx/conf/servers/servers_illstrator-plus-web.conf;
}

nginx server配置

1
2
3
4
5
6
7
8
9
10
11
12
13
server {
listen 80;
server_name api.xxx.com;
location / {
proxy_pass http://illstrator-plus-web/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_redirect off;
}
}

参考资料

关注公众号 尹安灿