Import Amazon S3 data with CloudFront-Simple Log Service(SLS)-阿里云帮助中心

Background

To meet requirements for data analysis, auditing, and monitoring, you can synchronize file data from Amazon S3 to Simple Log Service (SLS) for unified processing and retrieval. However, as data throughput grows, Amazon S3 outbound traffic costs can become a major operational expense. Especially in cross-cloud or large-file transfer scenarios, direct connections to S3 are not only more expensive but can also experience unstable transfer speeds due to network fluctuations during high-concurrency pulls.

This topic describes how to optimize the data transfer path using Amazon CloudFront. By configuring S3 as a CloudFront origin and switching the SLS data-pull endpoint to the CloudFront domain name, you can leverage its edge distribution capabilities instead of a direct connection. With CloudFront's more favorable tiered pricing and global network, you can reduce bandwidth costs while improving file download success rates and throughput, optimizing both cost and stability. This solution uses CloudFront only to fetch file data. File metadata is still retrieved using the original method, such as through SQS or a direct connection to S3. For enhanced security, you can use a CloudFront function to authenticate requests. This topic covers the following procedures: 1. Configure an authentication function; 2. Configure CloudFront; 3. Associate the function with a distribution; 4. Create an S3 import job.

Important

This is an original Alibaba Cloud document. All rights are reserved. This document may mention third-party products or company names to illustrate their integration with Alibaba Cloud services.

Configure authentication function

Step 1: Create a function

On the Functions page in CloudFront, create a function. For Runtime, select cloudfront-js-2.0.

Step 2: Edit and publish the function

Use the following code. Replace the token, build the function, and save it. After testing the function, publish it.

function handler(event) {
    var request = event.request;
    var headers = request.headers;

    // Define your token. Replace this with a strong, randomly generated string.
    var expectedToken = 'xxxxxx';

    // Verify the X-CF-Token header.
    if (!headers['x-cf-token'] || headers['x-cf-token'].value !== expectedToken) {
        return {
            statusCode: 403,
            statusDescription: 'Forbidden',
            body: {
                encoding: 'text',
                data: 'Access Denied'
            }
        };
    }

    // If the token is valid, forward the request.
    return request;
}

Click the Publish tab, and then click Publish function.

Configure CloudFront

Step 1: Create a distribution

On the Distributions page in CloudFront, create a distribution and choose a suitable pricing plan.

CloudFront offers Free, Professional, Business, and Enterprise pricing plans. Choose the plan that best fits your traffic volume and feature requirements.

Step 2: Name the distribution

In Distribution Options, set the distribution name. For distribution type, select Single website or app and then proceed to the next step.

Step 3: Configure S3 as origin

For origin type, select Amazon S3, and then click Browse S3 to select the target bucket.
Keep the default origin settings. For Cache settings, select Custom, and for the cache policy, select CachingDisabled.

Step 4: Enable security protections

Enable security protections as needed.

Step 5: Review and create the distribution

Review the configuration and create the distribution. Then wait for deployment to complete. Deployment is complete when the status changes to a timestamp.

Important

You will need the domain name from this page when you create the import job.

Associate function with distribution

Step 1: Select the distribution

In the distribution list, select the distribution created in Configure CloudFront. On the Behaviors tab, create a behavior.

Step 2: Associate the function

Scroll to the bottom of the page. For function type, select CloudFront Functions. In the viewer request row, select the function that you defined in Configure an authentication function, and then click Create.

Create an S3 import job

Step 1: Create the import job

Log on to the Log Service console. Click the name of your target Project to open it.
In the left-side navigation pane, click Job Management. On the Job Management page, click the Data Import tab, and then click Create Data Import Job.

Step 2: Start ingestion

In the dialog box that appears, click Integrate Now under S3-Data Import. Then, select the destination Logstore.

Step 3: Configure the job

Fill in the form. Turn on the Use CloudFront switch, and enter the CloudFront domain name and token that you obtained in the previous steps. Preview the settings, and then click Next.

Note

This section focuses on the CloudFront-related settings. For detailed instructions on all settings, see Import data from Amazon S3.