S3 (Simple Storage Solution) is a web-based object storage services introduced by Amazon in their public cloud service AWS. It is known for its scalability, data availability, security and performance. The main feature of that storage service is the reachability via ordinary web requests (https), which allows an easy usage in all kinds of applications.

Nowadays other provider of managed S3-compatible object storage are available just like OpenSource and proprietary software solutions to set up own private object storages.

S3App simplifies the access to S3Buckets with a provider independent web-based frontend which allows visualizing of the content of S3 buckets with an ordinary browser.

S3App is supporting all S3-compatible object storages. Nevertheless, if these are managed services like:

• Amazon Web Services S3
• Oracle Cloud Object Storage
• Alibaba Cloud OSS
• Backblaze B2
• DigitalOcean Spaces
• IBM Cloud Object Storage
• Linode Object Storage
• IONOS Cloud S3 Object Storage

and private (self-hosted) storages based on:

• Ceph (OpenSource)
• Cloudian (Proprietary)

You want your own private S3-compatible object storage based on Ceph, please get in touch with us.

Features of S3App

• Visualizing S3-Bucket content
• Downloading S3 buckets content
• Visualizing S3Bucket configuration (ACL, CORS, static Website, Permissions, …)
• Wide range of S3-providers like AWS, Oracle, IBM are directly usable
• Full support of multi-region providers. Buckets can be organized in different regions than the one the S3-Endpoint is mapped on.
• Role based authorization independently from the S3 provider access management
• Allow authentication in user groups (LDAP)
• User/Password and LDAP authentication independently from the S3 provider access management (a user needs no personal access or secret key)
• Easy to install and to configure with Python/PIP
• Kubernetes ready with an own HELM project

S3App URLs

The project contains code, build packages, container … . Below are the links to the different repositories.

S3App Concept

S3 Access Components

S3Provider

In S3App a S3Provider is a general service provider for an S3-compatible object storage. This could be a self-hosted storage or a managed storage. A provider offers a range of endpoints depend on the service regions the provider has (Multi-Region), like Europe (e.g., eu-central-1, eu-west-1), America (e.g., us-west-1), Asia (e.g., ap-east-1), …

In S3Aoo, major providers of managed object storages like AWS, Oracle, IBM are preconfigured with their endpoints.

You are able to add own providers. This is especially needed for complex self-hosted object storages based on Ceph or Cloudian.

S3Endpoint

An S3Endpoint is an endpoint provided by the S3Provider, doesn’t matter if it is in AWS, a Ceph or Cloudian storage.

In Multi-Region object storages, the endpoint depends on the region of the service provider. In principle, you only need a single S3Endpoint, mostly the “nearest by” endpoint to your application server, independently in which region your buckets are really located.

For example, if your applications are running on server in Germany, we recommend choosing an endpoint near to your server, like the endpoint for region eu-central-1.

In self-hosted environments with only one possible endpoint, you can add this endpoint directly to the endpoint. A S3Provider is not needed in this case.

S3Access

An S3Access contains the authentication credentials for the S3Provider. All providers need an Access Key and a Secret Key for authorization and authentication independently if these are managed or self-hosted providers.

In S3App a user with the view role does not need to know its access- and secrets keys the viewer is using to access their S3Buckets. A viewer is unable to analyse keys in S3App.

This allows that different users or a group of user can share the same S3Access.

Authentication and Authorization

User

A user has a username and password to authenticate against S3App. With the username and password, a user can authenticate against the database or via LDAP.

The user

• can be part of a user group,
• is linked directly or indirectly (via the user group) to one or more S3Accesses which contain the permissions for accessing S3Buckets and
• is linked directly or indirectly (via the user group) to a S3Role which contain the permissions in S3App (not in S3)

User Groups

In S3App a user group contains one or more users.

A group

• can be mapped onto a LDAP group. A separated user management in S3App is not needed in that case,
• is linked directly to an S3Access to allow the users in the group access to S3Buckets and

Roles

Roles contain the permissions of a user in S3App. In roles, you are not able to allow or to restrict access to S3Buckets.

A3App is currently providing the following pre-given roles:

S3Users: A user with the S3Users role can see the content of the S3 Buckets he/she has access to.

S3Manager: An S3Manager has additional permissions to an S3User. An S3Manager can see the setting of an S3Bucket like the Permission configuration, the ACL, Logging, …

Admin: It is the administrator role of S3App. He can manage S3Providers, S3Endpoints, S3Accesses, and can add Groups and Users. Usually an Admin has no own S3Buckets in access.

Installation

We provide two installation methods. A native installation with Python3/PIP which is working on every OS and an installation in Kubernetes via HELM.

Per default, S3App runs on a Sqlite database. But Sqlite is recommended only for a standalone installation for testing.

In production, you should use a PostgreSQL database for a better data persistence and to allow High Availability with more S3App nodes.

The web application S3App itself has no TLS-termination (“https”). We recommend using a Reverse Proxy like NGINX or Apache Web Server in front of the S3App web application for TLS termination.

The HELM project of S3App for Kubernetes contains all needed and recommended components, like a PostgreSQL or the Reverse Proxy. But you can also use separated applications.

A detailed description of the native installation you find in GitHub.

For the Kubernetes, you find the description in ArtifactHub.

Manual

Login

Username: Add your username

Password: Add your password

Sign In: Click for login. You get directly to the S3-dashboards

Dashboards

S3 Bucket Items

The dashboard is showing the content of one specific S3 Bucket and (Sub-)folder.

In that folder, you can step into a subfolder or download a file of a folder by clicking on it. Folders have a suffix “/”, files have additional metadata, like the size and the last modified entry.

Select Access: Accesses are more or less different providers or endpoints of S3 Buckets. Here you can select the provider of S3Buckets.

Select Bucket: Select the Buckets by name you have access to.

Show … entries: Select the amount of entries, you want to see in the list.

Prefix Search: S3 allows a prefix search. Add a prefix of the names to search the complete content of your current folder.

Download a file: To download a file, you can simply click on that file. The download begins in the background immediately.

Step through the folder: To get into a folder, you can simply click on that folder in the file / folder list. In the header, you find the path of that folder. You can choose the folder to get the folder up.

S3 Bucket Configuration

The dashboard S3 Bucket Configuration shows the configuration of your selected bucket.

Please be aware that not all functions are available for all providers (e.g., WebSite, logging, notifications, …) or are inactive for the bucket.

The metrics do not display the real usage of the Bucket. Aspects like versioning or additional hidden data added by the Provider are not reflected. The metrics only reflect the content you uploaded to the current active version.

Dashboard Menu

Up right you can click on your name to open the S3App menu. Depend on your role, you have different entries.

Edit User/Groups (Role Admin): Change the Users and User Groups of S3App.

Edit S3 (Role Admin): Change the accesses to the S3 providers, endpoints and map users or user group to that access.

S3 (Role S3Manager and S3User): See the bucket items (S3Items) or the bucket configuration (S3Buckets, only for Role S3Manager) . User (all): Get into your own profile to see your configuration or to change your password or to logout.

Profile and Reset Password

In the profile, you can see your setting and you can change your password.

Reset my password: Reset your own password.

In case you forgot your password, please inform the administrator of S3App. He is able to create a temporary dummy password for you.

Admin functions

A user with the Admin role can manage the users, user groups and S3 Endpoints for the user. An admin doesn’t need necessarily access to a bucket.

Listings

S3App is working with lists. In the list you have the choice to see, add, edit or to delete an entry.

You can reach the lists via the menu. All lists are looking the same and have similar functions for editing.

Add S3Provider

A S3Provider is a managed or unmanaged provider of an S3-compatible object storage. For your comfort, we added some providers like AWS, Oracle, … with their settings per default.

We recommend adding own S3Provider only, if they are supporting Multi-Region with endpoints for every region. A single cluster Ceph installation has only one single region. Then you can add the URL directly to the S3Endpoint.

To add an S3Provider, you have the following Parameter:

Name (mandatory): Your unique name of the S3Provider.

Endpoint Url Template: An URL template of the endpoints of that S3Provider. In Multi-Region object-storages, the endpoints of the regions only differ in the region part of the endpoint URL. For that you can add a <region> placeholder in that URL template, like for AWS s3.<region>.amazonaws.com.

Homepage Url: The URL with a link to the provider of that endpoint. This entry is optional.

S3 Endpoints: List of S3Endpoints currently using this S3Provider.

Add S3Endpoint

To add an S3Endpoint, you have the following Parameter:

Name (mandatory): Your unique name of the S3Endpoint.

S3 Provider: Choose an S3Provider if available. S3Provider are recommended if you use an S3Provider with Multi-Region functionality.

Url (mandatory): If you have a single cluster environment without Multi-Region, you can add the URL of the https endpoint as the host’s name directly. No S3Provider is needed in this case.

S3 Accesses: List of S3Accesses using this S3Endpoint. You can manage S3Access linked to this S3Endpoint directly.

S3 Default Region: The default region to use for this endpoint. We recommend using a region near to you even if the Buckets are located in different regions.

Trust Ca Bundle: Public Cloud provider are using signed certificates for the TLS termination (https). If you provide your own S3, this is not necessarily given. When the S3 Endpoint https access is secured with self-signed certificates, you must add the trusted CA-certificate here.

Add S3Access

To add an S3Access, you have the following Parameter:

Name (mandatory): Your unique name of the S3Access.

S3 Access Key (mandatory): The S3 access key to authenticate the access to the S3Buckets.

S3 Secret Key (mandatory): The S3 secret key to authenticate the access to the S3Buckets.

S3 Endpoint (mandatory): Endpoints linked with this access.

Users: List of users which have that S3Access directly linked.

Groups: List of groups which can use that S3Access.

Add user

To add a user, you have the following Parameter:

First Name (mandatory): First name of the user.

Last Name (mandatory): Last name of the user.

User Name (mandatory): A unique username for that user. It is used for authentication.

Is Active: You can disable a user before you want to remove it. When it is inactive the user cannot authenticate against S3App anymore.

Email (mandatory): The email address of the user.

Role (mandatory): The Role of the user.

(Confirm) Password (mandatory): The default password for the user.

S3 Accesses: The S3Accesses the user has access to.

Groups: The groups the user is part of.

Add group

To add a group, you have the following Parameter:

Name (mandatory): The unique name of the group.

Users: Users which are part of that group.

S3 Accesses: S3Accesses of that group.

Reset user password

All users can rest their password self-managed. If a user password gets lost, an admin with access to a S3App-Node can reset the password for the user:

flask --app s3app fab reset-password
The username [admin]: testuser
Password: 
Repeat for confirmation:
...
User testuser reset. 

Planned feature: It is planned that a user can reset its password self managed and vie EMail confirmation.

Performance Tuning

S3App is creating messages like

Task queue depth is 2

frequently. It shows the queue of requests currently not handled by S3App.

A low value for the queue is OK an can raise for a short time. A longer duration with an high value for the queue means a reduction in quality for the user of S3App.

If this happens, there are two positions to increase the performance:

1. Add more threads: The start of S3App has the parameter THREADS. It set the amount of thread per S3App server allowed to process requests. Increase this value if needed. Default is 4.
2. Add more nodes: When you use a load-balanced environment like in Kubernetes, you can add some more S3App-Nodes to your cluster of nodes.

A detailed description of the native installation and configuration you find in GitHub.

For the Kubernetes installation and configuration, you find the description here.

Are you interested? Let's arrange a call.