Skip to content

Commit a7d09c1

Browse files
ddelgrosso1gcf-owl-bot[bot]
ddelgrosso1
and
gcf-owl-bot[bot]
authored
feat: add transfer manager MPU sample, adjust defaults, prepare for GA (#2318)
* feat: add transfer manager MPU sample, adjust defaults, prepare for GA * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * adjust default values, remove experimental tags * fix wording * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
1 parent aadb3be commit a7d09c1

10 files changed

+130
-32
lines changed

‎README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/nodejs-storage/tre
131131
| Download Byte Range | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadByteRange.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadByteRange.js,samples/README.md) |
132132
| Download Encrypted File | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadEncryptedFile.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadEncryptedFile.js,samples/README.md) |
133133
| Download File | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadFile.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadFile.js,samples/README.md) |
134-
| Download a File in Chunks Utilzing Transfer Manager | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadFileInChunksWithTransferManager.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadFileInChunksWithTransferManager.js,samples/README.md) |
134+
| Download a File in Chunks With Transfer Manager | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadFileInChunksWithTransferManager.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadFileInChunksWithTransferManager.js,samples/README.md) |
135135
| Download File Using Requester Pays | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadFileUsingRequesterPays.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadFileUsingRequesterPays.js,samples/README.md) |
136136
| Download Folder With Transfer Manager | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadFolderWithTransferManager.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadFolderWithTransferManager.js,samples/README.md) |
137137
| Download Into Memory | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/downloadIntoMemory.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/downloadIntoMemory.js,samples/README.md) |
@@ -209,6 +209,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/nodejs-storage/tre
209209
| Upload Directory With Transfer Manager | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadDirectoryWithTransferManager.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadDirectoryWithTransferManager.js,samples/README.md) |
210210
| Upload Encrypted File | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadEncryptedFile.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadEncryptedFile.js,samples/README.md) |
211211
| Upload File | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadFile.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadFile.js,samples/README.md) |
212+
| Upload a File in Chunks With Transfer Manager | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadFileInChunksWithTransferManager.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadFileInChunksWithTransferManager.js,samples/README.md) |
212213
| Upload File With Kms Key | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadFileWithKmsKey.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadFileWithKmsKey.js,samples/README.md) |
213214
| Upload From Memory | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadFromMemory.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadFromMemory.js,samples/README.md) |
214215
| Upload Many Files With Transfer Manager | [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadManyFilesWithTransferManager.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadManyFilesWithTransferManager.js,samples/README.md) |

‎samples/README.md

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ objects to users via direct download.
5050
* [Download Byte Range](#download-byte-range)
5151
* [Download Encrypted File](#download-encrypted-file)
5252
* [Download File](#download-file)
53-
* [Download a File in Chunks Utilzing Transfer Manager](#download-a-file-in-chunks-utilzing-transfer-manager)
53+
* [Download a File in Chunks With Transfer Manager](#download-a-file-in-chunks-with-transfer-manager)
5454
* [Download File Using Requester Pays](#download-file-using-requester-pays)
5555
* [Download Folder With Transfer Manager](#download-folder-with-transfer-manager)
5656
* [Download Into Memory](#download-into-memory)
@@ -128,6 +128,7 @@ objects to users via direct download.
128128
* [Upload Directory With Transfer Manager](#upload-directory-with-transfer-manager)
129129
* [Upload Encrypted File](#upload-encrypted-file)
130130
* [Upload File](#upload-file)
131+
* [Upload a File in Chunks With Transfer Manager](#upload-a-file-in-chunks-with-transfer-manager)
131132
* [Upload File With Kms Key](#upload-file-with-kms-key)
132133
* [Upload From Memory](#upload-from-memory)
133134
* [Upload Many Files With Transfer Manager](#upload-many-files-with-transfer-manager)
@@ -718,7 +719,7 @@ __Usage:__
718719

719720

720721

721-
### Download a File in Chunks Utilzing Transfer Manager
722+
### Download a File in Chunks With Transfer Manager
722723

723724
Downloads a single file in in chunks in parallel utilizing transfer manager.
724725

@@ -2086,6 +2087,25 @@ __Usage:__
20862087

20872088

20882089

2090+
### Upload a File in Chunks With Transfer Manager
2091+
2092+
Uploads a single file in in chunks in parallel utilizing transfer manager.
2093+
2094+
View the [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadFileInChunksWithTransferManager.js).
2095+
2096+
[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-storage&page=editor&open_in_editor=samples/uploadFileInChunksWithTransferManager.js,samples/README.md)
2097+
2098+
__Usage:__
2099+
2100+
2101+
`node uploadFileInChunksWithTransferManager.js <BUCKET_NAME> <FILE_NAME> <CHUNK_SIZE>`
2102+
2103+
2104+
-----
2105+
2106+
2107+
2108+
20892109
### Upload File With Kms Key
20902110

20912111
View the [source code](https://github.com/googleapis/nodejs-storage/blob/main/samples/uploadFileWithKmsKey.js).

‎samples/downloadFileInChunksWithTransferManager.js

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,14 @@
1212
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1313
* See the License for the specific language governing permissions and
1414
* limitations under the License.
15-
* @experimental
15+
*
1616
*/
1717

1818
const path = require('path');
1919
const cwd = path.join(__dirname, '..');
2020

2121
// sample-metadata:
22-
// title: Download a File in Chunks Utilzing Transfer Manager
22+
// title: Download a File in Chunks With Transfer Manager
2323
// description: Downloads a single file in in chunks in parallel utilizing transfer manager.
2424
// usage: node downloadFileInChunksWithTransferManager.js <BUCKET_NAME> <FILE_NAME> <DESTINATION_FILE_NAME> <CHUNK_SIZE>
2525

‎samples/downloadFolderWithTransferManager.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1313
* See the License for the specific language governing permissions and
1414
* limitations under the License.
15-
* @experimental
15+
*
1616
*/
1717

1818
// sample-metadata:

‎samples/downloadManyFilesWithTransferManager.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1313
* See the License for the specific language governing permissions and
1414
* limitations under the License.
15-
* @experimental
15+
*
1616
*/
1717

1818
// sample-metadata:

‎samples/system-test/transfer-manager.test.js

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,16 @@ describe('transfer manager', () => {
8080
);
8181
});
8282

83+
it('should upload a file utilizing chunked upload', async () => {
84+
const output = execSync(
85+
`node uploadFileInChunksWithTransferManager.js ${bucketName} ${firstFilePath} ${chunkSize}`
86+
);
87+
assert.match(
88+
output,
89+
new RegExp(`${firstFilePath} uploaded to ${bucketName}.`)
90+
);
91+
});
92+
8393
it('should upload a directory', async () => {
8494
const output = execSync(
8595
`node uploadDirectoryWithTransferManager.js ${bucketName} ${resourcesPath}`

‎samples/uploadDirectoryWithTransferManager.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1313
* See the License for the specific language governing permissions and
1414
* limitations under the License.
15-
* @experimental
15+
*
1616
*/
1717

1818
// sample-metadata:

‎samples/uploadFileInChunksWithTransferManager.js

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
/**
2+
* Copyright 2023 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*
16+
*/
17+
18+
// sample-metadata:
19+
// title: Upload a File in Chunks With Transfer Manager
20+
// description: Uploads a single file in in chunks in parallel utilizing transfer manager.
21+
// usage: node uploadFileInChunksWithTransferManager.js <BUCKET_NAME> <FILE_NAME> <CHUNK_SIZE>
22+
23+
function main(
24+
bucketName = 'my-bucket',
25+
filePath = './local/path/to/file.txt',
26+
chunkSize = 32 * 1024 * 1024
27+
) {
28+
// [START storage_transfer_manager_upload_chunks_concurrently]
29+
/**
30+
* TODO(developer): Uncomment the following lines before running the sample.
31+
*/
32+
// The ID of your GCS bucket
33+
// const bucketName = 'your-unique-bucket-name';
34+
35+
// The path of file to upload
36+
// const fileName = 'path/to/your/file';
37+
38+
// The size of each chunk to be uploaded
39+
// const chunkSize = 32 * 1024 * 1024;
40+
41+
// Imports the Google Cloud client library
42+
const {Storage, TransferManager} = require('@google-cloud/storage');
43+
44+
// Creates a client
45+
const storage = new Storage();
46+
47+
// Creates a transfer manager client
48+
const transferManager = new TransferManager(storage.bucket(bucketName));
49+
50+
async function uploadFileInChunksWithTransferManager() {
51+
// Uploads the files
52+
await transferManager.uploadFileInChunks(filePath, {
53+
chunkSizeBytes: chunkSize,
54+
});
55+
56+
console.log(`${filePath} uploaded to ${bucketName}.`);
57+
}
58+
59+
uploadFileInChunksWithTransferManager().catch(console.error);
60+
// [END storage_transfer_manager_upload_chunks_concurrently]
61+
}
62+
63+
process.on('unhandledRejection', err => {
64+
console.error(err.message);
65+
process.exitCode = 1;
66+
});
67+
main(...process.argv.slice(2));

‎samples/uploadManyFilesWithTransferManager.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1313
* See the License for the specific language governing permissions and
1414
* limitations under the License.
15-
* @experimental
15+
*
1616
*/
1717

1818
// sample-metadata:

‎src/transfer-manager.ts

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -33,39 +33,39 @@ const packageJson = require('../../package.json');
3333

3434
/**
3535
* Default number of concurrently executing promises to use when calling uploadManyFiles.
36-
* @experimental
36+
*
3737
*/
38-
const DEFAULT_PARALLEL_UPLOAD_LIMIT = 2;
38+
const DEFAULT_PARALLEL_UPLOAD_LIMIT = 5;
3939
/**
4040
* Default number of concurrently executing promises to use when calling downloadManyFiles.
41-
* @experimental
41+
*
4242
*/
43-
const DEFAULT_PARALLEL_DOWNLOAD_LIMIT = 2;
43+
const DEFAULT_PARALLEL_DOWNLOAD_LIMIT = 5;
4444
/**
4545
* Default number of concurrently executing promises to use when calling downloadFileInChunks.
46-
* @experimental
46+
*
4747
*/
48-
const DEFAULT_PARALLEL_CHUNKED_DOWNLOAD_LIMIT = 2;
48+
const DEFAULT_PARALLEL_CHUNKED_DOWNLOAD_LIMIT = 5;
4949
/**
5050
* The minimum size threshold in bytes at which to apply a chunked download strategy when calling downloadFileInChunks.
51-
* @experimental
51+
*
5252
*/
5353
const DOWNLOAD_IN_CHUNKS_FILE_SIZE_THRESHOLD = 32 * 1024 * 1024;
5454
/**
5555
* The chunk size in bytes to use when calling downloadFileInChunks.
56-
* @experimental
56+
*
5757
*/
58-
const DOWNLOAD_IN_CHUNKS_DEFAULT_CHUNK_SIZE = 10 * 1024 * 1024;
58+
const DOWNLOAD_IN_CHUNKS_DEFAULT_CHUNK_SIZE = 32 * 1024 * 1024;
5959
/**
6060
* The chunk size in bytes to use when calling uploadFileInChunks.
61-
* @experimental
61+
*
6262
*/
6363
const UPLOAD_IN_CHUNKS_DEFAULT_CHUNK_SIZE = 32 * 1024 * 1024;
6464
/**
6565
* Default number of concurrently executing promises to use when calling uploadFileInChunks.
66-
* @experimental
66+
*
6767
*/
68-
const DEFAULT_PARALLEL_CHUNKED_UPLOAD_LIMIT = 2;
68+
const DEFAULT_PARALLEL_CHUNKED_UPLOAD_LIMIT = 5;
6969

7070
const EMPTY_REGEX = '(?:)';
7171

@@ -165,7 +165,7 @@ export class MultiPartUploadError extends Error {
165165
* Class representing an implementation of MPU in the XML API. This class is not meant for public usage.
166166
*
167167
* @private
168-
* @experimental
168+
*
169169
*/
170170
class XMLMultiPartUploadHelper implements MultiPartUploadHelper {
171171
public partsMap;
@@ -383,7 +383,7 @@ class XMLMultiPartUploadHelper implements MultiPartUploadHelper {
383383
* @hideconstructor
384384
*
385385
* @param {Bucket} bucket A {@link Bucket} instance
386-
* @experimental
386+
*
387387
*/
388388
export class TransferManager {
389389
bucket: Bucket;
@@ -400,7 +400,7 @@ export class TransferManager {
400400
* @property {string} [prefix] A prefix to append to all of the uploaded files.
401401
* @property {object} [passthroughOptions] {@link UploadOptions} Options to be passed through
402402
* to each individual upload operation.
403-
* @experimental
403+
*
404404
*/
405405
/**
406406
* Upload multiple files in parallel to the bucket. This is a convenience method
@@ -429,7 +429,7 @@ export class TransferManager {
429429
* const response = await transferManager.uploadManyFiles('/local/directory');
430430
* // Your bucket will now contain all files contained in '/local/directory' maintaining the subdirectory structure.
431431
* ```
432-
* @experimental
432+
*
433433
*/
434434
async uploadManyFiles(
435435
filePathsOrDirectory: string[] | string,
@@ -502,7 +502,7 @@ export class TransferManager {
502502
* @property {string} [stripPrefix] A prefix to remove from all of the downloaded files.
503503
* @property {object} [passthroughOptions] {@link DownloadOptions} Options to be passed through
504504
* to each individual download operation.
505-
* @experimental
505+
*
506506
*/
507507
/**
508508
* Download multiple files in parallel to the local filesystem. This is a convenience method
@@ -534,7 +534,7 @@ export class TransferManager {
534534
* const response = await transferManager.downloadManyFiles('test-folder');
535535
* // All files with GCS prefix of 'test-folder' have been downloaded.
536536
* ```
537-
* @experimental
537+
*
538538
*/
539539
async downloadManyFiles(
540540
filesOrFolder: File[] | string[] | string,
@@ -594,7 +594,7 @@ export class TransferManager {
594594
* to use when downloading the file.
595595
* @property {number} [chunkSizeBytes] The size in bytes of each chunk to be downloaded.
596596
* @property {string | boolean} [validation] Whether or not to perform a CRC32C validation check when download is complete.
597-
* @experimental
597+
*
598598
*/
599599
/**
600600
* Download a large file in chunks utilizing parallel download operations. This is a convenience method
@@ -618,7 +618,7 @@ export class TransferManager {
618618
* // Your local directory now contains:
619619
* // - "large-file.txt" (with the contents from my-bucket.large-file.txt)
620620
* ```
621-
* @experimental
621+
*
622622
*/
623623
async downloadFileInChunks(
624624
fileOrName: File | string,
@@ -645,7 +645,7 @@ export class TransferManager {
645645

646646
let start = 0;
647647
const filePath = options.destination || path.basename(file.name);
648-
const fileToWrite = await fsp.open(filePath, 'w+');
648+
const fileToWrite = await fsp.open(filePath, 'w');
649649
while (start < size) {
650650
const chunkStart = start;
651651
let chunkEnd = start + chunkSize - 1;
@@ -704,7 +704,7 @@ export class TransferManager {
704704
* See {@link https://cloud.google.com/storage/docs/xml-api/post-object-multipart#request_headers| Request Headers: Initiate a Multipart Upload}
705705
* @property {boolean} [autoAbortFailure] boolean to indicate if an in progress upload session will be automatically aborted upon failure. If not set,
706706
* failures will be automatically aborted.
707-
* @experimental
707+
*
708708
*/
709709
/**
710710
* Upload a large file in chunks utilizing parallel upload opertions. If the upload fails, an uploadId and
@@ -731,7 +731,7 @@ export class TransferManager {
731731
* // - "large-file.txt"
732732
* ```
733733
*
734-
* @experimental
734+
*
735735
*/
736736
async uploadFileInChunks(
737737
filePath: string,

0 commit comments

Comments
 (0)