Upload Big Files to Google Cloud Storage Bucket Services

Use Cases and Dissimilar Ways to get Files Into Google Cloud Storage

Including AppEngine with Firebase Resumable Uploads

For this article I volition intermission downwardly down a few different ways to interact with Google Cloud Storage (GCS). The GCP docs state the following means to upload your information: via the UI, via the gsutil CLI tool, or via JSON API in various languages. I'll become over a few specific utilise cases and approaches for the ingress options to GCS beneath.

i. upload form on Google App Engine (GAE) using the JSON api
Use case: public upload portal (small files)
2. upload course with firebase on GAE using the JSON api
Use example: public upload portal (big files), uploads within mobile app
iii. gsutil control line integrated with scripts or schedulers like cron
Employ case: backups/archive, integration with scripts, migrations
4. S3 / GCP compatible file direction programs such equally Cyberduck
Use case: deject storage management via desktop, migrations
5. Cloud role (GCF)
Employ case: Integration, changes in buckets, HTTP requests
half dozen. Cloud console
Use instance: cloud storage management via desktop, migrations

1. App Engine nodejs with JSON API for smaller files

You can launch a small nodejs app on GAE for accepting smaller files directly to GCS ~20MB pretty easily. I started with the nodejs GCS sample for GAE on the GCP github account here.

This is a dainty solution for integrating uploads around 20MB. Just remember the nginx servers behind GAE have a file upload limit. So if you try and upload something say around 50MB, yous'll receive an nginx error: ☹️

Nginx file upload error

You can try and upload the file size limit in the js file just however the web servers backside GAE will have a limit for file uploads. So, if you plan to create an upload form on App Engine, be sure to have a file size limitation in your UI.

Nodejs upload form for small files — I'll likely take this app down at some point.

ii. App Engine nodejs Firebase with JSON API and Resumable uploads for large files

Since the previous instance only works for smaller files, I wondered how can we solve for uploading larger files say 100MB or 1GB? I started with the nodejs app engine storage instance here.

After attempting to utilise resumable uploads in GCS API with TUS and failing I enlisted help from my friend Nathan @ www.incline.digital to help with another approach.

With the help of Nathan we integrated resumable uploads with firebase SDK. Code can be found here
https://github.com/mkahn5/gcloud-resumable-uploads.

Reference: https://firebase.google.com/docs/storage/web/upload-files

User interaction with the Firebase powered GAE Upload Form

While not very elegant with no status bar or annihilation fancy this solution does work for uploading large files from the web. 🙌🏻

Resumable File upload class on GAE — I'll likely take this app down at some bespeak.

GCS in the Firebase UI

three. gsutil from local or remote

gsutil makes it easy to copy files to and from deject storage buckets

Just make sure y'all accept the google deject sdk on your workstation or remote server (https://cloud.google.com/sdk/downloads), set project and authenticate and thats information technology.

          mkahnucf@meanstack-3-vm:~$            gsutil ls            
gs://artifacts.testing-31337.appspot.com/
gs://staging.testing-31337.appspot.com/
gs://testing-31337-public/
gs://testing-31337.appspot.com/
gs://us.artifacts.testing-31337.appspot.com/
gs://vm-config.testing-31337.appspot.com/
gs://vm-containers.testing-31337.appspot.com/          mkahnucf@meanstack-3-vm:~/nodejs-docs-samples/appengine/storage$            gsutil cp app.js gs://testing-31337-public                    Copying file://app.js [Content-Type=awarding/javascript]...          / [1 files][  2.7 KiB/  2.7 KiB]          Operation completed over ane objects/two.7 KiB.        

More details here.

gsutil makes information technology just easy to automate backup of directories, sync changes in directories, backup database dumps, and hands integrate with apps or schedulers for scripted file uploads to GCS.

Below is the rsync cron I have for my cloud storage bucket and the html files on my blog. This way I accept consistency between my GCS bucket and my GCE instances if I decide to upload a file via www or via GCS UI.

Using gsutil on GCP to backup and sync GCS files

          root@mkahncom-instance-group-multizone-kr5q:~# crontab -l          */2 * * * *            gsutil rsync -r /var/www/html gs://mkahnarchive/mkahncombackup                    */ii * * * *            gsutil rsync -r gs://mkahnarchive/mkahncombackup /var/www/html                  

4. Cyberduck (MacOS) or any application with an s3 interface

Bask an client ftp type experience with Cyberduck on MacOS for GCS.

Cyberduck has very squeamish oauth integration for connecting to the GCS API congenital into the interface.

After authenticating with oauth you can browse all of your buckets and upload to them via the cyberduck app. Prissy option to have for moving many directories or folders into multiple buckets.

More info on CyberDuck here.

five. Deject Office

You can also configure a Google Cloud Office (GCF) to upload files to GCS from a remote or local location. This tutorial beneath is just for uploading files in a directory to GCS. Run the deject function and it zips a local directory files and puts the zip into the GCS stage bucket.

Effort the tutorial:
https://cloud.google.com/functions/docs/tutorials/storage

          Michaels-iMac:gcf_gcs mkahnimac$            gcloud beta functions deploy helloGCS -stage-saucepan mike-kahn-functions -trigger-bucket mikekahn-public-upload            
Copying file:///var/folders/kq/5kq2pt090nx3ghp667nwygz80000gn/T/tmp6PXJmJ/fun.goose egg [Content-Blazon=application/zip]…
- [ane files][ 634.0 B/ 634.0 B]
Operation completed over 1 objects/634.0 B.
Deploying office (may take a while — up to 2 minutes)…washed.
availableMemoryMb: 256
entryPoint: helloGCS
eventTrigger:
eventType: providers/cloud.storage/eventTypes/object.alter
resource: projects/_/buckets/mikekahn-public-upload
latestOperation: operations/bWlrZS1rYWhuLXBlcnNvbmFsL3VzLWNlbnRyYWwxL2hlbGxvR0NTL1VFNmhlY1RZQV9j
name: projects/mike-kahn-personal/locations/us-central1/functions/helloGCS
serviceAccount: mike-kahn-personal@appspot.gserviceaccount.com
sourceArchiveUrl: gs://mike-kahn-functions/us-central1-helloGCS-wghzlmkeemix.zip
status: Fix
timeout: 60s
updateTime: '2017–05–31T03:08:05Z'        

Y'all tin also utilize cloud functions created to brandish saucepan logs. Below shows a file uploaded via my public upload form and deleted via the console ui. This could exist handy for pub/sub notifications or for reporting.

          Michaels-iMac:gcf_gcs mkahnimac$            gcloud beta functions logs read helloGCS                    LEVEL  NAME      EXECUTION_ID     TIME_UTC                 LOG          D      helloGCS  127516914299587  2017-05-31 03:46:19.412  Function execution started
I      helloGCS  127516914299587  2017-05-31 03:46:xix.502  File FLIGHTS BANGKOK.xlsx metadata updated.
D      helloGCS  127516914299587  2017-05-31 03:46:nineteen.523  Function execution took 113 ms, finished with status: 'ok'
D      helloGCS  127581619801475  2017-05-31 18:31:00.156  Function execution started
I      helloGCS  127581619801475  2017-05-31 xviii:31:00.379  File FLIGHTS BANGKOK.xlsx deleted.
D      helloGCS  127581619801475  2017-05-31 18:31:00.478  Function execution took 323 ms, finished with status: 'ok'        

Cloud Functions tin can come in handy for background tasks like regular maintenance from events on your GCP infrastructure or from activity on HTTP applications. Check out the how-to guides for writing and deploying cloud functions hither.

six. Cloud Panel UI

The UI works well for GCS administration. GCP even has a transfer service for files on S3 buckets on AWS or other s3 buckets elsewhere. Ane matter that is defective in the portal currently would be object lifecycle direction. This is overnice for automated archiving to coldline cheaper object storage for infrequently accessed files or files over a sure age in buckets. For at present y'all tin only modify object lifecycle via gsutil or via API. Like most GCP features they beginning at the function/API level and so make their mode into that portal (the way it should be IMO) and I'm fine with that. I expect object lifecycle rules to be implemented into the GCP portal at some point in the futurity. 😃

GCS UI

In summary I've used a few GCP samples and tutorials that are available to display to dissimilar ways to get files onto GCS. GCS is flexible with many ingress options that can exist integrated into systems or applications quite easily! In 2017 the employ cases for object storage are arable and GCP makes it easy to send and receive files in GCS.

Get out a comment for any interesting use cases for GCS that I may have missed or that we should explore. Thanks!

Check my blog for more than updates.

tanbobsely75.blogspot.com

Source: https://medium.com/google-cloud/use-cases-and-a-few-different-ways-to-get-files-into-google-cloud-storage-c8dce8f4f25a

Share this post