Alibaba Cloud HPC physical machines cannot access the public internet directly; they must use an ECS forward proxy for external connectivity.
This document guides users on how to set up a proxy server.
1. Identify IP Addresses
Users should first confirm the following IP addresses:
- ECS Public IP (not disclosed here, represented as XXX.XXX.XXX.XXX) and Internal IP (example: 10.10.10.10).
- HPC Physical Machine Internal IP (example: 10.239.23.4).
2. Log in to the ECS Bastion Host
Users can log in to the ECS using PuTTY (Windows environment) or the SSH command (Linux environment). Note that you must use the ECS public IP to log in.
ssh -l login_name XXX.XXX.XXX.XXX (ECS Public IP)
After successful login, you can log in to the HPC physical machine from the ECS bastion host using SSH:
ssh -l root 10.239.23.4 (HPC Physical Machine Internal IP)
3. Deploy Squid Proxy Server on the ECS Bastion Host
We choose Squid because it supports both HTTP and HTTPS services.
3.1 Install Squid
Open a new terminal and log in to the ECS bastion host.
Install directly using yum:
yum install squid
By default, the installation location is /usr/sbin/squid.
3.2 Edit Squid Configuration File
Open the /etc/squid/squid.conf file with root privileges. First, comment out all unnecessary internal network addresses, add a line above them to include your HPC physical machine’s IP address, comment out ports that do not need to be open, and keep only ports 80 and 443. Then, add a line: access_log /var/log/squid/access.log to record access logs.
The modified configuration looks like this:
......
acl localnet src 10.239.23.4 # Add a line here with your HPC physical machine IP address
#acl localnet src 10.0.0.0/8 # RFC1918 possible internal network, commented out
#acl localnet src 172.16.0.0/12 # RFC1918 possible internal network, commented out
#acl localnet src 192.168.0.0/16 # RFC1918 possible internal network, commented out
#acl localnet src fc00::/7 # RFC 4193 local private network range, commented out
#acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines, commented out
acl SSL_ports port 443
acl Safe_ports port 80 # Open HTTP service port
#acl Safe_ports port 21 # FTP can be opened depending on requirements
acl Safe_ports port 443 # Open HTTPS service port
#acl Safe_ports port 70 # gopher, commented out
#acl Safe_ports port 210 # wais, commented out
#acl Safe_ports port 1025-65535 # unregistered ports, commented out
#acl Safe_ports port 280 # http-mgmt, commented out
#acl Safe_ports port 488 # gss-http, commented out
#acl Safe_ports port 591 # filemaker, commented out
#acl Safe_ports port 777 # multiling http, commented out
acl CONNECT method CONNECT
......
# And finally deny all other access to this proxy
http_access deny all # Deny access to everything except what is allowed above
# Squid normally listens to port 3128
http_port 3128 # Default Squid listening port; can be changed to another port
# Added
access_log /var/log/squid/access.log # Add a line here to monitor access records
Save the file.
3.3 Start Squid
Run with root privileges: sudo service squid start
Check the Squid running status:
sudo service squid status
squid (pid 31659) is running...
3.4 Configure ECS Firewall
To save traffic and ensure security, configure firewall rules on the ECS to block all requests to port 3128 except those from the HPC physical machine.
Steps are as follows:
3.4.1 Enable Firewall
CentOS 6: service iptables start
CentOS 7: systemctl start firewalld
3.4.2 Add Firewall Rules
First, allow the HPC physical machine IP address (example: 10.239.23.4, modify according to actual conditions) to access port 3128:
iptables -I INPUT -s 10.239.23.4 -p TCP --dport 3128 -j ACCEPT
Port 3128 must match the port setting in the Squid configuration file in section 3.2.
Then, deny all TCP connections to port 3128:
iptables -A INPUT -p TCP --dport 3128 -j DROP
Save the iptables settings:
service iptables save
Verify if the rules are effective:
iptables -L -n
You should see two new rules added:
target prot opt source destination
ACCEPT tcp -- 10.239.23.4 0.0.0.0/0 tcp dpt:3128
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:3128
4. Configure Proxy on the HPC Physical Machine
Return to the HPC physical machine terminal to configure the proxy.
The simplest way is to use environment variables. Assuming the ECS internal IP is 10.10.10.10 (users need to replace this with the real ECS internal IP), execute:
export http_proxy=http://10.10.10.10:3128
export https_proxy=http://10.10.10.10:3128
Note: The proxy server port setting here must match the listening port in the squid.conf on the ECS bastion host, and it must be the ECS internal address.
You can also add the above statements to /etc/profile or ~/.bashrc to automatically configure the proxy server when logging into the HPC physical machine.
5. Test
Test using wget, git clone, and yum install on the HPC physical machine. Results are as follows:
5.1 Test HTTP Access
wget http://www.cmake.org/files/v3.3/cmake-3.3.1.tar.gz
--2015-09-29 18:12:52-- http://www.cmake.org/files/v3.3/cmake-3.3.1.tar.gz
Connecting to 120.26.218.226:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 6577869 (6.3M) [application/x-gzip]
Saving to: ‘cmake-3.3.1.tar.gz’
100%[========================================================================================================================================>] 6,577,869 28.7KB/s in 3m 55s
2015-09-29 18:16:47 (27.4 KB/s) - ‘cmake-3.3.1.tar.gz’ saved [6577869/6577869]
You can check the /var/log/squid/access.log file to find the download log above:
1443530801.311 7144 10.239.23.4 TCP_MISS/200 132972 GET http://www.cmake.org/files/v3.3/cmake-3.3.1.tar.gz - DIRECT/66.194.253.19 application/x-gzip
5.2 Test HTTPS Access
wget https://codeload.github.com/gflags/gflags/tar.gz/v2.1.2
--2015-09-29 18:18:13-- https://codeload.github.com/gflags/gflags/tar.gz/v2.1.2
Connecting to 120.26.218.226:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 95716 (93K) [application/x-gzip]
Saving to: ‘v2.1.2’
100%[========================================================================================================================================>] 95,716 78.7KB/s in 1.2s
2015-09-29 18:18:17 (78.7 KB/s) - ‘v2.1.2’ saved [95716/95716]
You can check the /var/log/squid/access.log file to find the download log above:
1443531073.042 11238 10.239.23.4 TCP_MISS/200 101394 CONNECT codeload.github.com:443 - DIRECT/192.30.252.145 -
5.3 Test Git Clone
git clone https://github.com/gflags/gflags.git
Cloning into 'gflags'...
remote: Counting objects: 1772, done.
remote: Total 1772 (delta 0), reused 0 (delta 0), pack-reused 1772
Receiving objects: 100% (1772/1772), 1.27 MiB | 493.00 KiB/s, done.
Resolving deltas: 100% (1013/1013), done.
You can check the /var/log/squid/access.log file to find the download log above:
1443531119.620 29512 10.239.23.4 TCP_MISS/200 1356342 CONNECT github.com:443 - DIRECT/192.30.252.129 -
5.4 Test Yum
# yum install openssl
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.aliyuncs.com
* extras: mirrors.aliyuncs.com
* updates: mirrors.aliyuncs.com
Resolving Dependencies
--> Running transaction check
---> Package openssl.x86_64 1:1.0.1e-42.el7.9 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================================================================================
Package Arch Version Repository Size
==================================================================================================================================================================================
Installing:
openssl x86_64 1:1.0.1e-42.el7.9 updates 711 k
Transaction Summary
==================================================================================================================================================================================
Install 1 Package
Total download size: 711 k
Installed size: 1.5 M
Is this ok [y/d/N]: y
Downloading packages:
openssl-1.0.1e-42.el7.9.x86_64.rpm | 711 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
authconfig-6.2.8-9.el7.x86_64 has missing requires of /usr/bin/openssl
Installing : 1:openssl-1.0.1e-42.el7.9.x86_64 1/1
Verifying : 1:openssl-1.0.1e-42.el7.9.x86_64 1/1
Installed:
openssl.x86_64 1:1.0.1e-42.el7.9
Complete!
You can check the /var/log/squid/access.log file to find the yum installation log above:
1443522032.861 129 10.239.23.4 TCP_MISS/200 728129 GET http://mirrors.aliyuncs.com/centos/7/updates/x86_64/Packages/openssl-1.0.1e-42.el7.9.x86_64.rpm - DIRECT/10.143.34.200 application/x-redhat-package-manager
Through these tests, it is verified that the HPC physical machine can now access the public internet via the ECS forward proxy.