缘起
今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求，由于4月份自学考试临近，一直在备考就拖着没总结记录下，开发联调过程中也出现过一些奇葩的问题，最近人刚从考试缓过来顺手记录一下。
S3对象存储的基本概念
S3是什么？
Amazon S3(Simple Storage Service)对象存储出现得比较早且使用简单的RESTful API，于是成为了对象存储服务(Object Storage Service，OSS)业内的标准接口规范。
S3的逻辑模型
如下图，我们可以把S3的存储空间想象成无限的，想存储一个任意格式的文件到S3服务中，只需要知道要把它放到哪个桶(Bucket)中，它的名字（Object Id）应该是什么。
按图中的模型，可简单理解为S3是由若干个桶（Bucket）组成，每个桶中包含若干个不同标识的对象（Object），还有就是统一的访问入口(RESTful API)，这样基本就足够了。
Minio客户端方式操作S3
详细API文档：https://min.io/docs/minio/linux/developers/java/API.html
以下代码异常处理做了简化，真实使用时请注意捕获异常做处理。
引入依赖
Maven:
<dependency> <groupId>io.minio</groupId> <artifactId>minio</artifactId> <version>8.5.2</version> </dependency> Gradle:
dependencies { implementation("io.minio:minio:8.5.2") } 初始化客户端
private static final String HTTP_PROTOCOL = "http"; private MinioClient minioClient; private String endpoint = "http://192.168.0.8:9200"; private String accessKey = "testKey"; private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); try { // url上无端口号时，识别http为80端口，https为443端口 int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort(); boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true; //@formatter:off this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security) .credentials(accessKey, secretKey).build(); //@formatter:on // 忽略证书校验，防止自签名证书校验失败导致无法建立连接 this.minioClient.ignoreCertCheck(); } catch (Exception e) { e.printStackTrace(); } } 建桶
public boolean createBucket(String bucket) { try { minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); return false; } return true; } 删桶
public boolean deleteBucket(String bucket) { try { minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build()); logger.info("删除桶[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true; } 判断桶是否存在
public boolean bucketExists(String bucket) { try { return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); return false; } } 上传对象
public void upload(String bucket, String objectId, InputStream input) { try { //@formatter:off minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId) .stream(input, input.available(), -1) .build()); //@formatter:on } catch (Exception e) { e.printStackTrace(); } } 下载对象
提供两个下载方法，一个将输入流返回，另一个用参数输出流写出
public InputStream download(String bucket, String objectId) { try { return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } return null; } public void download(String bucket, String objectId, OutputStream output) { //@formatter:off try (InputStream input = minioClient.getObject( GetObjectArgs.builder().bucket(bucket).object(objectId).build())) { IOUtils.copyLarge(input, output); } catch (Exception e) { e.printStackTrace(); } //@formatter:on } 删除对象
public boolean deleteObject(String bucket, String objectId) { //@formatter:off try { minioClient.removeObject(RemoveObjectArgs.builder() .bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } //@formatter:on return true; } 判断对象是否存在
public boolean objectExists(String bucket, String key) { //@formatter:off try { // minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在 minioClient.statObject(StatObjectArgs.builder() .bucket(bucket).object(key).build()); } catch (Exception e) { return false; } //@formatter:on return true; } 完整代码
import java.io.InputStream; import java.io.OutputStream; import java.net.MalformedURLException; import java.net.URL; import org.apache.tomcat.util.http.fileupload.IOUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import io.minio.BucketExistsArgs; import io.minio.GetObjectArgs; import io.minio.MakeBucketArgs; import io.minio.MinioClient; import io.minio.PutObjectArgs; import io.minio.RemoveBucketArgs; import io.minio.RemoveObjectArgs; import io.minio.StatObjectArgs; public class S3MinioClientDemo { private static final Logger logger = LoggerFactory.getLogger(S3MinioClientDemo.class); private static final String HTTP_PROTOCOL = "http"; private MinioClient minioClient; private String endpoint = "http://192.168.0.8:9200"; private String accessKey = "testKey"; private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); try { // url上无端口号时，识别http为80端口，https为443端口 int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort(); boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true; //@formatter:off this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security) .credentials(accessKey, secretKey).build(); //@formatter:on // 忽略证书校验，防止自签名证书校验失败导致无法建立连接 this.minioClient.ignoreCertCheck(); } catch (Exception e) { e.printStackTrace(); } } public boolean createBucket(String bucket) { try { boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build()); if (found) { logger.info("桶名[{}]已存在", bucket); return false; } minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); } return true; } public boolean deleteBucket(String bucket) { try { minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build()); logger.info("删除桶[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true; } public boolean bucketExists(String bucket) { try { return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build()); } catch (Exception e) { e.printStackTrace(); return false; } } public void upload(String bucket, String objectId, InputStream input) { try { //@formatter:off minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId) .stream(input, input.available(), -1) .build()); //@formatter:on } catch (Exception e) { e.printStackTrace(); } } public InputStream download(String bucket, String objectId) { try { return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } return null; } public void download(String bucket, String objectId, OutputStream output) { //@formatter:off try (InputStream input = minioClient.getObject( GetObjectArgs.builder().bucket(bucket).object(objectId).build())) { IOUtils.copyLarge(input, output); } catch (Exception e) { e.printStackTrace(); } //@formatter:on } public boolean objectExists(String bucket, String objectId) { //@formatter:off try { // minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在 minioClient.statObject(StatObjectArgs.builder() .bucket(bucket).object(objectId).build()); } catch (Exception e) { return false; } //@formatter:on return true; } public boolean deleteObject(String bucket, String objectId) { //@formatter:off try { minioClient.removeObject(RemoveObjectArgs.builder() .bucket(bucket).object(objectId).build()); } catch (Exception e) { e.printStackTrace(); } //@formatter:on return true; } public void close() { minioClient = null; } } Amazon S3 SDK方式操作S3
官方API文档：https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
这里由于项目上提供的SDK和文档都是1.x的，这里就暂时只提供1.x的代码
引入依赖
Maven:
<dependency> <groupId>com.amazonaws</groupId> <artifactId>aws-java-sdk-s3</artifactId> <version>1.11.300</version> </dependency> Gradle:
dependencies { implementation 'com.amazonaws:aws-java-sdk-s3:1.11.300' } 初始化客户端
private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class); private AmazonS3 s3client; private String endpoint = "http://192.168.0.8:9200"; private String accessKey = "testKey"; private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); String protocol = endpointUrl.getProtocol(); int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort(); ClientConfiguration clientConfig = new ClientConfiguration(); clientConfig.setSignerOverride("S3SignerType"); clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase())); // 禁用证书检查，避免https自签证书校验失败 System.setProperty("com.amazonaws.sdk.disableCertChecking", "true"); // 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题 System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true"); AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey); // 创建 S3Client 实例 AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig); s3client.setEndpoint(endpointUrl.getHost() + ":" + port); s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build()); this.s3client = s3client; } 建桶
public boolean createBucket(String bucket) { String bucketName = parseBucketName(bucket); try { if (s3client.doesBucketExist(bucketName)) { logger.warn("bucket[{}]已存在", bucketName); return false; } s3client.createBucket(bucketName); } catch (Exception e) { e.printStackTrace(); } return true; } 删桶
public boolean deleteBucket(String bucket) { try { s3client.deleteBucket(bucket); logger.info("删除bucket[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true; } 判断桶是否存在
public boolean bucketExists(String bucket) { try { return s3client.doesBucketExist(bucket); } catch (Exception e) { e.printStackTrace(); } return false; } 上传对象
public void upload(String bucket, String objectId, InputStream input) { try { // 创建文件上传的元数据 ObjectMetadata meta = new ObjectMetadata(); // 设置文件上传长度 meta.setContentLength(input.available()); // 上传 s3client.putObject(bucket, objectId, input, meta); } catch (Exception e) { e.printStackTrace(); } } 下载对象
public InputStream download(String bucket, String objectId) { try { S3Object o = s3client.getObject(bucket, objectId); return o.getObjectContent(); } catch (Exception e) { e.printStackTrace(); } return null; } public void download(String bucket, String objectId, OutputStream out) { S3Object o = s3client.getObject(bucket, objectId); try (InputStream in = o.getObjectContent()) { IOUtils.copyLarge(in, out); } catch (Exception e) { e.printStackTrace(); } } 删除对象
public boolean deleteObject(String bucket, String objectId) { try { s3client.deleteObject(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; } return true; } 判断对象是否存在
public boolean existObject(String bucket, String objectId) { try { return s3client.doesObjectExist(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; } } 完整代码
import java.io.InputStream; import java.io.OutputStream; import java.net.MalformedURLException; import java.net.URL; import org.apache.tomcat.util.http.fileupload.IOUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.amazonaws.ClientConfiguration; import com.amazonaws.Protocol; import com.amazonaws.auth.AWSCredentials; import com.amazonaws.auth.BasicAWSCredentials; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3Client; import com.amazonaws.services.s3.S3ClientOptions; import com.amazonaws.services.s3.model.ObjectMetadata; import com.amazonaws.services.s3.model.S3Object; /** * S3对象存储官方SDK实现 * * @author ZhangChenguang * @date 2023年2月2日 */ @SuppressWarnings("deprecation") public class S3SdkDemo { private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class); private AmazonS3 s3client; private String endpoint = "http://192.168.0.8:9200"; private String accessKey = "testKey"; private String secretKey = "testSecretKey"; public void init() throws MalformedURLException { URL endpointUrl = new URL(endpoint); String protocol = endpointUrl.getProtocol(); int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort(); ClientConfiguration clientConfig = new ClientConfiguration(); clientConfig.setSignerOverride("S3SignerType"); clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase())); // 禁用证书检查，避免https自签证书校验失败 System.setProperty("com.amazonaws.sdk.disableCertChecking", "true"); // 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题 System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true"); AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey); // 创建 S3Client 实例 AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig); s3client.setEndpoint(endpointUrl.getHost() + ":" + port); s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build()); this.s3client = s3client; } public boolean createBucket(String bucket) { try { s3client.createBucket(bucket); } catch (Exception e) { e.printStackTrace(); } return true; } public boolean deleteBucket(String bucket) { try { s3client.deleteBucket(bucket); logger.info("删除bucket[{}]成功", bucket); } catch (Exception e) { e.printStackTrace(); return false; } return true; } public boolean bucketExists(String bucket) { try { return s3client.doesBucketExist(bucket); } catch (Exception e) { e.printStackTrace(); } return false; } public void upload(String bucket, String objectId, InputStream input) { try { // 创建文件上传的元数据 ObjectMetadata meta = new ObjectMetadata(); // 设置文件上传长度 meta.setContentLength(input.available()); // 上传 s3client.putObject(bucket, objectId, input, meta); } catch (Exception e) { e.printStackTrace(); } } public InputStream download(String bucket, String objectId) { try { S3Object o = s3client.getObject(bucket, objectId); return o.getObjectContent(); } catch (Exception e) { e.printStackTrace(); } return null; } public void download(String bucket, String objectId, OutputStream out) { S3Object o = s3client.getObject(bucket, objectId); try (InputStream in = o.getObjectContent()) { IOUtils.copyLarge(in, out); } catch (Exception e) { e.printStackTrace(); } } public boolean existObject(String bucket, String objectId) { try { return s3client.doesObjectExist(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; } } public boolean deleteObject(String bucket, String objectId) { try { s3client.deleteObject(bucket, objectId); } catch (Exception e) { e.printStackTrace(); return false; } return true; } public void close() { s3client = null; } } 遇到的问题
1、bucket名称必须是小写，不支持下划线
- 处理方式：写方法转换下bucket名称，将大写转小写，将下划线替换为中划线。
2、minio客户端下载非官方S3存储的文件时，如果响应头的Content-Length与实际文件大小不符，会导致minio客户端包装的okhttp3报错
报错信息：
Caused by: java.net.ProtocolException: unexpected end of stream at okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430) ~[okhttp-3.14.9.jar:?] at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[okhttp-3.14.9.jar:?] at okio.RealBufferedSource$1.read(RealBufferedSource.java:447) ~[okio-1.17.2.jar:?] at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83) ~[nr.file-2.5.7.jar:?] at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395) ~[nr.file-2.5.7.jar:?] ... 122 more 抓包发现问题的图：
最终换成了S3官方SDK可用了。
PS：客户现场部署的S3是浪潮公司提供的，如果现场遇到这个情况，就不要固执去找对方对线了，完全没用。。
总结
S3存储的基本操作就记录到这里了，由于没有S3存储就没尝试官方SDK的V2版本，由于这些代码是总结时从业务代码里抽取出来的，可能会有点问题，但大体思路已经有了。
希望对读者有所用处，觉得写得不错和有帮到你，欢迎点个赞，您的支持就是我的鼓励！
![洛谷 P11345 [KTSC 2023 R2] 基地简化 题解](http://www.itfaba.com/wp-content/themes/kemi/timthumb.php?src=http://www.itfaba.com/wp-content/themes/kemi/img/random/1.jpg&w=218&h=124&zc=1)
