Object Subsystem (ObjectNode)¶
The object subsystem provides S3-compatible object storage interface. It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.
Structure¶
The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.
ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster which it belong to directly without any volume-mount operation.
Features¶
- Provides object storage interface compatible with native Amazon S3 SDKs.
- Fusion storage expose two type interface (POSIX and S3-compatible).
- Stateless and high scalability
Semantic Transform¶
Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.
| POSIX | Object Storage |
|---|---|
Volume |
Bucket |
Path |
Key |
Example:
Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.
Authentication¶
The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with volume creation. The AccessKey is a 16-character string unique in the entire ChubaoFS cluster.
Authentication keys owned by volume and stored with volume view (volume topology) by Resource Manager (Master). User can fetch it by using administration API, see Get Volume Information at Volume
Invisible Temporary Data¶
In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.
Object Key Conflict (Important)¶
The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.
But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.
The object key like this is conflict in ChuBaoFS.
Supported S3-Compatible APIs¶
Bucket APIs¶
| API | Reference |
|---|---|
HeadBucket |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html |
GetBucketLocation |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html |
Object APIs¶
Multipart Upload APIs¶
| API | Reference |
|---|---|
CreateMultipartUpload |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html |
ListMultipartUploads |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html |
AbortMultipartUpload |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html |
CompleteMultipartUpload |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html |
ListParts |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html |
UploadPart |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html |
UploadPartCopy |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html |
Supported SDKs¶
Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.
| Name | Language | Link |
|---|---|---|
| AWS SDK for Java | Java |
https://aws.amazon.com/sdk-for-java/ |
| AWS SDK for JavaScript | JavaScript |
https://aws.amazon.com/sdk-for-browser/ |
| AWS SDK for JavaScript in Node.js | JavaScript |
https://aws.amazon.com/sdk-for-node-js/ |
| AWS SDK for Go | Go |
https://docs.aws.amazon.com/sdk-for-go/ |
| AWS SDK for PHP | PHP |
https://aws.amazon.com/sdk-for-php/ |
| AWS SDK for Ruby | Ruby |
https://aws.amazon.com/sdk-for-ruby/ |
| AWS SDK for .NET | .NET |
https://aws.amazon.com/sdk-for-net/ |
| AWS SDK for C++ | C++ |
https://aws.amazon.com/sdk-for-cpp/ |
| Boto3 | Python |
http://boto.cloudhackers.com |