LLM-Clean Copyright Information (MaxCompute)

Supported computing resources

MaxCompute

Algorithm

The component removes copyright or other comment information from text in the following two steps:

1. First, the algorithm checks if the text contains a string that matches the regular expression '/\\*[^*]*\\*+(?:[^/*][^*]*\\*+)*/' (comment characters).

   • If a matching string is found, the algorithm checks whether it contains the copyright keyword. If so, it deletes the entire string and returns the result. Otherwise, it returns the text unchanged.

   • If no match for the regular expression is found, the algorithm proceeds to Step 2.

2. The algorithm splits the text by line breaks. The algorithm then iterates through each line to check if it starts with a comment marker, such as //, #, or --. Once a matching line is found, the algorithm continues to count consecutive comment lines until a non-comment line is found. Finally, the algorithm deletes the consecutive block of comments from the text and returns the result.

Both preceding steps detect only the first matching comment block, which means the component only checks the header of the text. The component does not process the remaining content. The following example shows the before and after.

(function(window, angular, undefined) {
'use strict';
angular.module('angularSpinner', [])
.factory('usSpinnerService', ['$rootScope', function ($rootScope) {
    var config = {};
    config.spin = function (key) {
        $rootScope.$broadcast('us-spinner:spin', key);
    };
}]);
})(window, window.angular);

Configure the component

In Machine Learning Designer, add the LLM-Clean Copyright Information (MaxCompute) component to a pipeline and configure its parameters in the pane on the right.