AWS FAQ: Common Architecture and Troubleshooting Questions

Users often encounter various issues when using AWS. This article summarizes common architecture and troubleshooting questions in a Q&A format, hoping to be helpful. Issues specific to the China (Beijing and Ningxia) regions will be specifically noted.

Q: How to improve S3 performance?

A: S3 separates the index from the actual data. By default, all objects in a bucket are located in the same partition. If there are too many objects and the request rate is not high, S3 will automatically partition based on the key name. However, if the request rate is too high, it can exhaust the I/O of the S3 partition, triggering limits and potentially returning a 503 Slow Down error.

Try to utilize hashing or reverse ordering to randomize the S3 prefix. If read/write requests are excessively high, you can submit an S3 partition request to AWS or use an exponential backoff algorithm to increase the retry wait time. Nevertheless, you must still follow the S3 key name randomization principle. You can partition already randomized objects under a specific directory.

For example, the .log object prefixes in the log directory of an S3 bucket have been randomized:

S3://bucket/log/A0B1.log
S3://bucket/log/C2D3.log
S3://bucket/log/E4F5.log

When submitting a request, in addition to the request rate, you need to fill in the pattern, indicating how randomization is performed in the log/ directory. The backend can then partition after S3://bucket/log/.

S3 does not limit bandwidth. However, do not stress test S3. The actual transmission rate depends on the network between the client and S3. If there are many client requests, you can use CDN for acceleration.

Q: There are many small files; how to upload them to S3 efficiently?

A: If possible, try to compress them into large files before uploading. On one hand, this reduces the overhead of establishing and tearing down TCP connections for numerous small file uploads. On the other hand, although S3 does not charge for upload traffic, requests are charged.

Q: Need to sync files daily from local to S3. There are currently very many existing files (millions). How to improve efficiency?

A: The aws s3 sync command can sync files from local to S3, syncing only the changed parts. However, sync first needs to list LIST S3 to compare. If the number of files is huge, listing all objects takes a very long time and may even hang.

Solution: Sync only the changed directories. For example, sync by date directories.

aws s3 sync /data/2018/03/13 s3://bucket/data/2018/03/13

If daily changed data is distributed across different directories, you can record these changes in DynamoDB and only sync the changed parts during synchronization.

Q: Does S3 support resume transfer?
A: Resume transfer is implemented by dividing into multiple ranges based on content-length. Unfinished parts can be retained and continued next time.

S3 supports multi-part transfer. It involves three steps:

1. Initiate Multipart Upload: Initialize the task.
2. Parts Upload: Split the file into multiple chunks, each with the same upload ID and different part numbers. Different chunks are transmitted separately, regardless of order.
3. Complete Multipart Upload: After all chunks are completed, merge them into an S3 object. General programs, such as AWS CLI, if some parts are incomplete and timeout after retries, will call Abort Multipart Upload, deleting all parts already uploaded to S3, terminating the task.

S3 does not provide a direct resume transfer API; you can add logic in your program to implement it. If some parts in the second stage never finish uploading, the third stage’s Complete/Abort will not be called. Thus, the transferred parts remain in S3 until Complete/Abort.

Q: When an application downloads S3 files, it checks MD5 to prevent tampering during transmission, but large files always fail.

A: Large file transfers in S3 use multi-part segmentation. MD5 is calculated for each segment separately. Finally, when the task is complete, the MD5 of all segments is calculated again to obtain the S3 Etag value.

The application only compared the MD5 value of a single segment, which was inconsistent with the S3 Etag, causing an error.

Q: In the scenario of mobile app uploads to S3, what is the difference between S3 Pre-signed URL and Cognito methods?

A: When a mobile client uploads, it needs to generate a Pre-signed URL on the server and return it to the client. The client uses this temporary URL to upload without credentials. Actually, during upload, it uses the IAM user permissions when the server generated the URL, effectively uploading as that IAM user. The limitation of this method is that the pre-signed URL does not know the size of the entire uploaded object, so S3 multi-part cannot be used, resulting in low efficiency for large file transfers.

In a Cognito environment, the mobile client first performs custom or federated authentication. After verification, it exchanges tokens with Cognito. Then, the client obtains temporary credentials from STS using the Cognito token to access S3. With this method, the client can use normal APIs, including multi-part.

Cognito in the China region currently lacks user pool functionality and does not support Google/Facebook authentication. You can manage users yourself, e.g., LDAP/AD. For small file uploads, pre-signed URLs generally meet requirements.

Q: What is S3 eventual consistency like?

A: For newly created S3 objects, read-after-write consistency is adopted; they can be accessed immediately after upload.

For objects overwritten or operated on via HEAD/GET, eventual consistency is adopted. If accessed immediately after overwriting, it might be new content or previous content, but eventually, it will be new content.

Q: My account’s S3 ELB access logs have bucket policies authorizing other accounts to view them, why can’t other accounts still access them?

A: ELB access logs are objects uploaded by the AWS system account to your S3 bucket after authorization. The log file owner is the AWS account, and your account was added to the Object ACL during upload, allowing you to access. However, the file owner is not your account, and other accounts were not added to the Object ACL. Even if the bucket policy allows other accounts to access, they still cannot.

Solution: Add the canonical ID of other accounts to the object ACL. The ID can be obtained via AWS CLI command:

aws s3api list-buckets
{
    "Owner": {
        "ID": "633eb40d3277ec23db4bc5a394dc363f48e41c85358c5618f0de3df382855242"
},

Note: For objects uploaded to another account’s bucket, if the object ACL is not authorized to the bucket owner, the bucket owner cannot download or modify the ACL, only delete.

Q: Why doesn’t the lifecycle setting take effect?

A: After setting the S3 lifecycle, it starts working at 0:00 UTC the next day. All objects, based on creation time or the time they became non-current versions, if conditions are met, will be transitioned to Glacier/Standard IA types or deleted. In cases with too many objects, this process may take longer.

If it never takes effect, check if the lifecycle rule contains spaces, or use s3api head-object to see if conversion is occurring.

If converting to Standard IA type, the S3 object size must be below 128KB.

Q: Browser access to S3 object URL results in 401 Unauthorized error. (China)

A: Please complete ICP filing and open ports 80/443/8080 for the account.

Q: Downloading S3 objects via the browser in the AWS console results in Unexpected IP error. (China)

A: Downloading S3 objects via the console generates a special pre-signed URL with the x-amz-expect-ip parameter, restricting the IP that can use this URL. When the client returns to upload using this URL, if the IP changes (including company exit IP changes, using proxies, etc.), an Unexpected IP error will occur.

Q: How to sync Global overseas S3 to China S3? (China)

A: AWS Global and China use different accounts and cannot directly sync via S3 Cross-region replication.

To sync automatically, consider S3 Notification + Lambda. When S3 objects change, notifications trigger Lambda to automatically call functions to copy objects. Note that this method is affected by the Internet network status between China and overseas and does not support large files well.

Another method: S3 Notification + SQS. S3 notifications are sent to SQS message queues. EC2 runs programs to periodically fetch messages from SQS. If there are updates, the overseas EC2 downloads overseas S3 objects, copies them to the China region EC2, and then uploads them to China S3 from the China region EC2. Although this method is more complex, it fully utilizes the optimized network between AWS cross-region EC2s and can use the S3 multi-part feature, making it more suitable for large file transfers.

Q: EC2 instance status check failed, stop stuck.

A: EC2 instance status, if 0/2, indicates underlying hardware failure. You can stop (stop) the instance and then start (start), migrating the virtual machine to another physical host. If the stop operation gets stuck, force stop again. If it still doesn’t work, open a technical support case. For C3, C4, C5, M3, M4, M5, R3, R4, T2, or X1 type instances, it is recommended to enable EC2 automatic recovery. When underlying hardware fails, it can recover itself.

Please note: Instance store data will be lost. Use EBS for important data and perform snapshot backups.

If the status check is 1/2, it means the underlying hardware is normal, but the operating system has failed, requiring self-troubleshooting combined with log information.

Q: What is the bandwidth of EC2 instances?

A: EC2 instance network performance varies by type (low, medium, high). Due to different network conditions from clients to AWS, testing is needed to obtain actual values, e.g., using iperf tools. Some types, like C4, can reach 10Gb network bandwidth internally between EC2 instances, but internet network performance still requires actual testing.

Some types support enhanced networking to improve network performance. This feature may require installing drivers and enabling modules in the OS.

Q: Accessing a website on EC2 is slow.

A: Access speed is affected by the server, network, and client throughout the process.

At the server level, CPU/memory/IO resource usage, OS kernel parameters, and application optimization all affect user experience. Monitor CPU/IO/network via Cloudwatch, check memory and processes on the OS, check OS and application logs, and use tools like strace to trace calls if necessary.

At the network level, from client to server, packets pass through multiple routers, and carriers may experience congestion or routing issues. Use bidirectional traceroute/mtr tools to check which network routes might have problems. Confirm if the network issue is universal by checking if users in other regions face similar issues. If it’s carrier network congestion or the client accessing complex networks, this exceeds AWS’s scope to resolve.

At the client level, check own CPU/memory resources. Generally, dynamic HTTP requests with complex database queries or pages downloading many images/videos/css/js are slower than simple HTML pages. Try another client with the same request for testing.

Q: EC2 was stopped or terminated, but not by me.

A: AWS will not shut down user EC2 instances, even due to underlying hardware failure. Check if other services like Autoscaling/Cloudformation triggered the operation. Open Cloudtrail to see API call records.

Q: Does EC2 support broadcasting?

A: EC2 does not support broadcast or multicast, only unicast. Some applications, like LVS keepalived, need to be set to unicast mode.

Q: EC2 security group allows access from another security group, but it doesn’t work.

A: Allowing another security group in the security group requires using the private IP during connection. If using the public IP, the security group must be set to the public IP CIDR.

Additionally, internal EC2 access should preferably use private IPs to reduce network transmission costs.

Q: Created a CentOS instance with a 10GB EBS root disk, but df command only shows 8GB.

A: The default CentOS root disk partition table is 8GB. You need to manually change the partition table and extend the filesystem.

Q: Created an instance with a 3TB root disk, but the instance terminated immediately after startup.

A: Windows uses MBR partitioning, supporting a maximum 2TB root disk. Linux MBR has the same limitation.

Solution: Use an EBS smaller than 2TB for the root disk and place data on separately added EBS. Linux can also use GPT partitioning, which is not limited to 2TB.

Q: CentOS multi-network card configuration not taking effect.

A: Non-Amazon Linux systems like CentOS lack certain network installation packages and require manual configuration of policy routing. This ensures packets received from a specific network card are sent back through the same card.

Q: Created a C4 type CentOS instance, showing 1/2 check status, unable to log in.

A: C3/C4 type instances have enhanced networking enabled by default. Some non-Amazon Linux systems may lack the ixgbev enhanced network driver, preventing normal startup.

Solution: Install the ixgbevf enhanced network driver.

Q: How to use CentOS in the China region? (China)

A: The China region currently has no official CentOS. You can import local VM images via VM Import, or import AMIs from AWS overseas regions to the China region via dd image files.

For security reasons, unofficial community CentOS is not recommended.

Q: Why isn’t EBS reaching expected IOPS, but is being throttled?

A: Besides IOPS, EBS has a maximum throughput limit (GP2 160MB/s, IO1 500MB/s). This relates to IO size. Check if Cloudwatch throughput metrics have reached the upper limit.

<pAdditionally, continuous small IOs are merged into single IOs.

Q: Created an EBS, but couldn’t find the required instance when attaching.

A: EBS must be in the same Availability Zone (AZ) as the EC2 instance to attach.

Q: How to automatically create snapshots for EBS?

A: Method 1: Create scheduled tasks to periodically execute AWS CLI commands for snapshots.

Method 2: Use Cloudwatch Events + Lambda for serverless automatic execution.

Q: Accessing ELB, some requests are normal, others timeout.

A: Public ELBs must be in a public subnet within the VPC (i.e., having an IGW) to be accessible from the Internet. If the subnet containing the ELB includes private subnets, the ELB will create nodes in the private subnet. When accessing from the Internet, DNS resolves to nodes in the private subnet, making those requests unreachable.

Q: Backend server logs show all request source IPs are ELB. How to get the real client IP?

A: For HTTP listeners, enable the X-Forwarded-For feature on backend servers.

For TCP listeners, ELB needs to enable forward protocol and add specific headers.

Q: What algorithm does ELB/ALB use to forward requests to backend instances?

A: ELB TCP listener ports use round-robin algorithm; HTTP listener ports use least outstanding requests algorithm.

ALB uses round-robin algorithm.

If there is only one healthy backend instance, all requests are forwarded to this instance. If there are no healthy backend instances, requests are forwarded to all registered but unhealthy instances.

Q: ELB is not controlled by the customer. How to troubleshoot server errors?

A: Client requests returning 5XX errors can be preliminarily judged as ELB or backend server issues based on HTTP codes, combined with Cloudwatch HTTP 5XX, ELB 5XX, Backend Connection Errors, and other monitoring metrics.

If it’s a 504 Gateway Timeout, open ELB access logs and perform packet capture analysis on both client and backend servers. Generally, if the backend web server enables keep-alive, the timeout time should be greater than the ELB idle timeout.

Q: How to choose subnets for NLB?

A: NLB must be in a public subnet. Backend target instances can be in public or private subnets. Note: If backend instances are in private subnets, NAT must be configured to allow packets to route back to the client. This differs from ELB/ALB.

Q: Application accessing ELB sometimes shows Unknown host resolution error.

A: ELB automatically scales up or down based on traffic, changing IPs. DNS servers need to update changes promptly. ELB DNS TTL is 60 seconds; after ELB IP changes, DNS records update within 60 seconds. However, due to client caching, if not updated, resolution fails.

Default Java DNS TTL is permanent. Restarting Java temporarily solves the problem. It is recommended to modify Java DNS TTL to 60 seconds.

Q: How to estimate average request/response traffic for ELB when applying for warm-up or static IP?

A: Example: A 200KB webpage is unlikely to pass all 200KB of traffic through ELB in a single request/response. A more likely distribution is: 20KB HTML, 30KB Javascript/CSS, 150KB for 10 15KB images, totaling 12 requests. Thus, the average request/response traffic through ELB is not 200KB but 17KB (200/12=16.67). If static content and downloads are optimized in S3 or CDN, this ELB average traffic will be even smaller.

Q: How to retrieve archived files in Glacier by filename?

A: Glacier archive objects have an archive ID for retrieval. However, the archive ID is a string of characters, not a filename. You can use a database (e.g., DynamoDB) to record the correspondence between filenames and archive IDs. Or, write the filename in the description of the archive object. In cases with many archived files, using a database is recommended to improve efficiency.

Q: Does it support high availability architecture?

A: Regardless of the gateway type, a gateway VM must be installed locally or on EC2 to act as the storage gateway between local and AWS. This gateway is singular; the VM itself does not support high availability architecture.

Q: File share type gateway has poor transfer speed.

A: Factors affecting performance include: network, S3, storage gateway, client.

After mounting the NFS client, all read requests go to the gateway first. If the gateway cache has data, it returns directly to the client.

Write requests also arrive at the gateway first, written to the gateway’s local storage disk, then asynchronously copied to S3.

S3 does not limit transmission bandwidth. Generally, the local network to Internet exit bandwidth only affects the gateway copying data to S3.

For the client, transfer speed is relative to the client to local storage gateway. Client settings and local gateway performance are the decisive factors.

Windows 2008 supports NFSv3; Windows 2012/2016 supports NFSv4. V4 cache is much larger than V3, significantly impacting NFS performance.

The gateway server needs sufficient CPU processing power and memory. Upload buffer minimum 150GB, Cache suggested to be 1.1 times the upload buffer. On the physical machine hosting the storage gateway, when creating VM disk files, separate the physical disks for the system and cache/upload buffers, preferably using SSDs.

Q: How to make an IAM user logging into the AWS console only able to start or stop certain EC2 instances?

A: An IAM user logging into the AWS console either lists all EC2 instances or lists nothing. Because DescribeInstances permission does not support resource-level control, it cannot individually control permissions for certain resources (EC2 instances, VPCs, etc.). For permissions not supporting resource levels, the Resource section in the policy can only use * to represent all resources.

StartInstances/StopInstances support resource-level permissions, allowing definition of instance IDs or control via other conditions.

The following IAM policy allows listing all EC2 instances and statuses, and only performing start/stop operations on instances tagged Group=test.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": "arn:aws-cn:ec2:cn-north-1:794767850066:instance/*",
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/Group": "test"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*"
            ],
            "Resource": "*"
        }
    ]
}

Besides tags, you can also directly define instance IDs or other matching conditions to control stop/start permissions.

Reference:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-supported-iam-actions-resources.html

Q: Testing on physical machines and AWS RDS yields significantly different results for similar models and the same test methods. Why?

A: A prerequisite for test comparison is consistent environments. Besides server processing power and test methods, storage performance and database parameters also affect results.

If RDS is restored from a snapshot, EBS storage undergoes a warming-up process, needing to read data blocks in EBS first to achieve optimal performance. It is recommended to use newly created RDS instead of restoring from snapshots to avoid warming-up issues with fresh EBS.

EBS IOPS also have an impact. Monitor via Cloudwatch to ensure EBS IOPS or throughput haven’t reached the upper limit.

Database parameters greatly impact performance, e.g., innodb_flush_log_at_trx_commit. Ensure database parameters are consistent during test comparisons.

Q: Accessing applications in different AZs connecting to RDS shows significant latency differences.

A: To achieve high availability, deploy applications to EC2s in different AZs. RDS is located in a specific AZ. When accessing RDS in different AZs from EC2, there is slight network latency. If the application interacts multiple times between EC2 and RDS, this slight latency is amplified.

For businesses with high network latency requirements, optimize the application to avoid multiple database accesses in a single transaction. If the application cannot be adjusted, try placing application servers in the same AZ as RDS, but this poses high availability risks, requiring a trade-off between performance and availability.

Q: Many RDS commands execute with permission errors.

A: RDS is a managed service with many mechanisms to ensure database high availability. Aspects involving the OS or database底层 (e.g., system tables) generally cannot be modified. Common database operations can be performed via predefined stored procedures.

For example, killing a mysql process:

CALL mysql.rds_kill(processID);

Other operations, such as modifying database parameters, can be done via parameters in parameter groups.

Reference: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.MySQL.CommonDBATasks.html

Q: Can RDS MySQL master-slave replication select specific tables?

A: Master-slave replication between RDS and external MySQL can select tables via external MySQL’s /etc/my.cnf, replicate-ignore-db or replicate-do-db.

Internal RDS replication currently does not support table selection.

Q: RDS shows disk capacity exhausted, but there doesn’t seem to be that much data.

A: Besides database table data, indexes, logs, and temporary space also occupy storage. MySQL logs can occupy up to 20% or 10GB of storage space. Some complex queries may use significant temporary storage. You can set MySQL binlog retention time and whether to enable general log.

Q: RDS MySQL database restart prompts Out of memory.

A: Memory exhaustion is related to database parameters. max_connections defines max connections = memory bytes / 12582880. If this parameter is modified beyond limits, it may cause memory exhaustion. The monitoring mechanism detects the issue and automatically restarts the database.

Q: RDS MySQL master-slave replication latency keeps increasing.

A: Master and slave instance types should be consistent; otherwise, processing power differences can cause lag with large data volumes.

Check master and slave EBS IOPS and throughput to see if they’ve reached limits.

Check if the slave has lock operations.

View slave status: show slave status G; Check if errors occurred during replication causing stagnation. If so, skip errors and retry.

MySQL 5.7 allows parallel replication to improve sync efficiency. Enable this feature for large data volumes.

slave_parallel_type=LOGICAL_CLOCK
slave_parallel_workers>1

Q: RDS is set to public access. How to access RDS from EC2 within the same VPC without going through the public internet?

A: RDS domain names are generated when creating RDS. Applications need to access the RDS domain name. If EC2 and RDS are in the same VPC, DNS resolves to the intranet IP, connecting via private network.

Q: Why can’t RDS MySQL restore data via point-in-time recovery?

A: RDS MySQL needs to enable binlog. Recovery to a previous point in time is achieved via snapshot + binlog.

If automatic snapshot function is not enabled, binlogs are not saved, making point-in-time recovery impossible.

Additionally, only InnoDB tables can use this feature. MyISAM is not transaction-based and may lose data upon restart or database failure. Unless specifically required, it is recommended to use the InnoDB storage engine.

Q: How to perform database migration?

A: MySQL: Use full Mysqldump + incremental replication. However, during the period of exporting data via mysqldump and replicating to the target database, the master must be set to read-only to prevent new data writes.

Oracle: Dataguard or GoldenGate. But this requires specific Oracle Enterprise Edition or separate licensing.

AWS DMS Service: Online database synchronization. The master does not need to be set to read-only. During synchronization, the transit node reads log changes and replays them when syncing to the target database. Even if the database is in a private subnet, DMS can access the database and set it as source or target. DMS supports migration between different databases, including those supported by RDS and Redshift.

AWS DMT Tool: The China region has not yet launched the DMS service. You can apply for the DMT AMI, create an EC2 instance with the DMT graphical migration tool.

Q: DynamoDB has pre-configured many read/write capacities, but throttling errors occur with little usage.

A: DynamoDB pre-configures read/write capacities per table. DynamoDB has partitions, and each partition has limited resources. When pre-configured read/write capacities or data volume exceed a certain level, DynamoDB automatically partitions. Based on the primary key partition key hash value, it determines the partition for each item. If specific items have huge read/write volumes falling into a single partition, exhausting its I/O resources, throttling occurs.

Example: Setting 10,000 read capacity and 4,000 write capacity for a table might require a total of 8 partitions. Calculated based on preset capacity, each partition’s read capacity upper limit is 1250. If hot data reading is too frequent, reaching 1250 causes errors.

Another possibility: Previously set large read/write capacities, leading to automatic partitioning. Later reduced read/write capacities, but partitions did not decrease, allocating less read/write capacity per partition, making hot partition data throttling more likely.

Solutions:

• Increase read/write capacity.
• Avoid excessive hot data access.
• Add ElastiCache or DynamoDB DAX caching before DynamoDB to reduce read/write pressure.

Q: How to backup DynamoDB in the China region?

A: DynamoDB has backup and restore features in overseas regions, but not yet in the China region. You can use DynamoDB’s built-in CSV export feature or third-party tools to export data into files.

Q: How to monitor Elasticache Redis performance metrics?

A: Cloudwatch has the following metrics to watch:

• • CPUUtilization: CPU utilization. For cache services like Redis, CPU utilization is generally low.
  • SwapUsage: Swap partition. Used when memory is insufficient. However, occasional small swap usage might also be caused by certain OS actions. If other metrics like BytesUsedForCache, Evictions are normal, no need to worry too much.
  • Evictions: Number of keys evicted due to exceeding max memory limit. This is an important metric. Check BytesUsedForCache metric; if almost all allocated memory is used, it indicates insufficient memory.

BytesUsedForCache: Memory allocated to Redis. Note that some memory is reserved for backup or replication operations. It is possible to see that a Redis instance type selected with 16GB memory can only use up to 12GB. Newly created Redis reserves 25% memory by default. Check the reserved-memory and reserved-memory-percent parameters.

CurrConnections: Current client connections. If abnormal compared to before, check the application. Redis is single-threaded; clients generally manage connections via connection pools to improve efficiency.

FreeableMemory: Available memory at the OS level. A Linux mechanism is to use buffer/cache as much as possible; much memory is used for caching, so available memory is generally low. If available memory is low but other Redis metrics are normal, don’t worry.

Q: How to persist Elasticache Redis?

A: AOF (Append Only Files) stores data on local disk. This feature was disabled by default in version 2.8 and earlier, and cannot be modified in version 3.2 and above. It is recommended to use replication groups instead of AOF. Local disks are located on instance store, used for storing temporary data. If underlying hardware fails, local disk data, including AOF, will be lost. Replication groups are located in multiple AZs, automatically replicating data to read replicas, achieving automatic failover.

Q: If an Elasticache Redis replication group fails, does the application need adjustment?

A: If a read replica fails, a new read replica is automatically started to replace the failed node. The application needs to modify read requests to the new read replica. Read/write requests to the master node do not need modification.

If the master node fails, a read replica is automatically promoted to the new master node, and DNS points to the new IP. For the application, read/write requests still point to the master node DNS, requiring no modification. Read requests need modification, including to the new read replica.

Q: How to set Redis username/password login?

A: Elasticache Redis does not support username/password login. You can allow access only from trusted IPs via security groups and network ACLs.

Q: How to set Redis to allow access from the public internet?

A: Elasticache Redis is designed for internal access; DNS resolves to private addresses within the VPC. Accessing from the public internet affects performance due to network latency and is insecure.

If public internet access is absolutely necessary, you can set up an EC2 instance as NAT within the VPC, add iptables rules to access a specific port on the EC2 (e.g., 6379) from the public internet, and map it via NAT port mapping to the internal Redis port. Please note that this public internet traffic is not encrypted; choose this method cautiously.

Q: Elasticache Redis has cluster and non-cluster modes. How to choose?

A: Both modes support master-slave implementation for high availability. In cluster mode, each shard has a replication group, including one master node and multiple read replicas.

The difference between the two modes lies in scalability.

In non-cluster mode, there is only one master node. If memory is insufficient, you can only upgrade the node type. Read replicas max out at 5.

Cluster mode provides partitioning. If the master node has heavy write pressure, use cluster mode for partitioning, up to 15, writing data to multiple nodes for better scalability. For read requests, cluster mode can have up to 15*5=75 read replicas.

Q: How to improve concurrent COPY performance when importing large amounts of Gzip CSV from S3 to Redshift?

A: Redshift COPY performance is affected by the following factors:

• Parallel transfer. Split data into multiple files; a single COPY command can write multiple files in parallel to the same table. Note: This feature does not apply to multiple COPY commands writing to the same table; you must wait for the previous COPY to complete before continuing.
• Table compression. By default, COPY auto-compresses empty tables. Compression consumes resources, sacrificing some COPY performance for smaller storage space. A trade-off between performance and storage is needed. COMPUPDATE OFF (or FALSE) disables auto-compression.
• Table analysis. By default, the COPY command performs analysis after loading data into an empty table. Analysis runs the ANALYZE command to build and select optimal plan statistical metadata, also consuming resources. During import, adding STATUPDATE OFF (or FALSE) to the COPY command prevents updating statistics.
• Sort key. If the table defines a sort key (sort key), COPY can import data more efficiently and improve query performance.
• Distribution. When Redshift imports data, it stores each row of data on different compute nodes according to the distribution style. This is also crucial for subsequent query performance.
• File compression. Multiple csv gzip files placed directly in S3 need decompression when importing to Redshift, affecting performance. If they are all small compressed files, decompress them before uploading to S3, then copy to Redshift.

Q: Redshift creates joint views for multiple tables. Tables have created SORT KEY, but query performance is low.

A: Redshift sort keys are COMPOUND by default. If query conditions use all columns included in this sort key, query efficiency will be high. Conversely, if query conditions do not include the sort key, or only include part of the COMPOUND SORT KEY, query efficiency decreases.

Querying views requires traversing all data; views create temporary tables and scan these temporary tables. Even if source tables set sort keys, views do not use them, reducing query efficiency.

Solutions:

• Modify the sort key to the most needed sorting column, and use all columns of the sort key in queries, not just specifying part of the sort key. If only some columns are needed for optimized sorting, use INTERLEAVED SORT KEY statement when creating the table.
• Create only one table, importing daily data into this Redshift table via COPY command. Single table queries can directly use SORT KEY optimization.
• If other tables need JOIN operations, set the correct DISTKEY to avoid unnecessary inter-node data transmission.
• If table sorting is very messy or some space is marked as deleted but not reclaimed, run the vacuum command to reclaim space and perform sorting.

Q: Redshift cannot run large Queries in parallel, resulting in out of memory.

A: Below are some Redshift statistical information.

skew_rows represents the ratio of max rows to min rows in different slices. These tables are severely skewed, indicating improper allocation, consuming more memory, and in severe cases, exhausting memory resources. Also, some tables use date-type fields as dist keys, but date types are not suitable.

These issues consume large amounts of memory.

Solution:

Rebuild severely skewed tables, choosing appropriate DIST KEYs, preferably columns most associated with other tables.

Q: Can Redshift scale online?

A: Redshift can modify node count to achieve online scaling. The modification process involves creating a new Redshift cluster and copying data to it. If data volume is large, modification may take a long time. You can also consider creating a snapshot of the original cluster, restoring a new cluster from the snapshot, and then renaming it. This might save time.

Q: Why does accessing API Gateway result in 403 Forbidden error? (China)

A: Due to Chinese policy reasons, using API Gateway in China requires submitting an application to whitelist the account.

Q: Does SNS support mobile message push? (China)

A: The China region currently does not support mobile message push functionality or SMS text messaging. SNS can send messages to Lambda or SQS message queues; you can develop your own program for mobile message push.

Q: EMR data: How to choose between storing in HDFS or S3?

A: HDFS runs on EC2, and both are within the AWS internal network. Accessing S3 does not have a significant network latency difference compared to HDFS.

Compared to HDFS, storing data in S3 offers more benefits:

• S3 has 11 nines durability, virtually no data loss.
• S3 has infinite capacity, while HDFS is limited by local storage capacity, limiting future scalability.
• If data needs to be migrated to a new cluster, data located in S3 persists, whereas HDFS requires data migration.

Q: EMR resources insufficient, how to scale?

A: EMR can change node count. If data volume is large, it takes a long time, even days. Once modification starts, it cannot be stopped. If a large amount of data is stored in S3, you can rebuild EMR, even using custom AMIs, without losing data.

Q: EMR disk capacity insufficient.

A: Applications on EMR should preferably use S3 instead of HDFS where possible, avoiding occupying local storage.

/mnt /mnt1 These local directories are generally used for storing temporary data and logs. Sometimes temporary data is not cleaned up, causing these directories to fill the disk. Regular cleaning or running custom Cloudwatch disk usage monitoring scripts to send automatic notifications when disk space is low is needed.

Q: How to merge multiple small files in S3 into large files?

A: s3distcp runs MR tasks to merge multiple files from S3 or HDFS. For example:

s3-dist-cp --src s3://support-billing/ --dest s3://pingaws-support/billing/ --groupBy='.*(794767850066*?).*'