API调用实现华为云OBS对象的调用
对象存储由于具有高空间、低费用、可直接URL调用访问等特点，是最常见的存储方式之一。主流的云厂商都有相关的云服务，AWS的s3、Azure Blob stroage、GCP cloud stroage、华为阿里的OBS。在开启匿名访问后，任何人都可以通过对应的URL访问到对应的对象，所以出于安全原因，很多会通过ak/sk的方式生成调用的 signature，并结合expires过期时间来保证链接的盗用。本篇就使用huaweicloud OBS的API 生成对应的对象下载或调用链接。
一、Signature生成
我这里选用了 URL中携带签名的方式，根据官方文档内容，其实比较关键的第一步是拼出 stringtosign 的内容。
1StringToSign =
2 HTTP-Verb + "\n" +
3 Content-MD5 + "\n" +
4 Content-Type + "\n" +
5 Expires + "\n" +
6 CanonicalizedHeaders + CanonicalizedResource;
接下来算签名，步骤如下：
1、构造请求字符串(StringToSign)。
2、对第一步的结果进行UTF-8编码。
3、使用SK对第二步的结果进行HMAC-SHA1签名计算。
4、对第三步的结果进行Base64编码。
5、对第四步的结果进行URL编码，得到签名。
对应的代码如下：
1hashed = hmac.new(yourSecretAccessKeyID.encode('UTF-8'), canonical_string.encode('UTF-8'),hashlib.sha1)
2encode_canonical = binascii.b2a_base64(hashed.digest())[:-1].decode('UTF-8')
二. python实现
官方提供了JAVA实现的代码，相较于JAVA，我更喜欢用 python 来实现这种简单功能，具体代码如下：
1import sys
2import hashlib
3import hmac
4import binascii
5#from datetime import datetime,timedelta,fromtimestamp
6import datetime
7import urllib.request
8IS_PYTHON2 = sys.version_info.major == 2 or sys.version < '3'
9
10
11def get_str_timestamp(date):
12 # date = datetime.datetime.now()
13 return int(date.timestamp())
14
15yourSecretAccessKeyID = 'xxxxxxxxxxxxxxx'
16httpMethod = "GET"
17contentmd5 = ""
18#contentType = "application/octet-stream"
19contentType = ""
20# "date" is the time when the request was actually generated
21# now = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
22now = datetime.datetime.utcnow()
23plus = datetime.timedelta(days=2)
24dateplus = now + plus
25date = dateplus.strftime('%a, %d %b %Y %H:%M:%S GMT')
26
27date_stamp = get_str_timestamp(dateplus)
28
29print(date)
30print(date_stamp)
31#canonicalizedHeaders = "x-obs-acl:private\n"
32canonicalizedHeaders = ""
33CanonicalizedResource = "/ctslogstorage/CloudTraces/la-south-2/2023/09/15/system/ECS/CloudTrace_la-south-2_2023-09-15T15-46-20Z_5bfdd257091735a3.json.gz"
34canonical_string = httpMethod + "\n" + contentmd5 + "\n" + contentType + "\n" + str(date_stamp) + "\n" + canonicalizedHeaders + CanonicalizedResource
35if IS_PYTHON2:
36 hashed = hmac.new(yourSecretAccessKeyID, canonical_string, hashlib.sha1)
37 encode_canonical = binascii.b2a_base64(hashed.digest())[:-1]
38else:
39 hashed = hmac.new(yourSecretAccessKeyID.encode('UTF-8'), canonical_string.encode('UTF-8'),hashlib.sha1)
40 encode_canonical = binascii.b2a_base64(hashed.digest())[:-1].decode('UTF-8')
41print(encode_canonical)
42
43url = 'https://ctslogstorage.obs.la-south-2.myhuaweicloud.com/CloudTraces/la-south-2/2023/09/15/system/ECS/CloudTrace_la-south-2_2023-09-15T15-46-20Z_5bfdd257091735a3.json.gz?AccessKeyId=yourakinformation&Expires=' + str(date_stamp) + '&Signature=' + urllib.request.quote(encode_canonical)
44
45print(url)
这里是参考了Header中携带签名里的代码进行修改的。注意和Header中携带签名不同的是，这里的时间和header中使用的在生成stringtosign时是不同的：
- Header中携带签名使用的是 15 Sep 2023 09:20:09 GMT 这样的格式；
- URL中携带时，使用的expires时间是1532779451这样的格式（相较1970-1-1后的秒数）。
这里还有一个坑一定要注意（这个也是在读取了JAVA样例代码后发现的），最后生成的这个signature一定要URL串化的，注意看图上的最后一个等号在调用时是不是变成了%3D，如果里面有加号，会变成%2B，这个地方如果不改是不会成功的，会报错 SignatureDoesNotMatch 。
由于我这里读取的是存到OBS里的CTS审计日志，其默认是以json.gz格式存在OBS里的，对应的读取方法如下：
1import requests
2
3r = requests.get(url)
4with open("file.json.gz", "wb") as f:
5 f.write(r.content)
6
7import gzip
8import json
9
10# 读取json.gz文件
11with gzip.open('file.json.gz', 'r') as f:
12 content = f.read()
13
14json_data = json.loads(content)
15
16# 打印json数据
17print(json_data)
三、Java实现方式
由于JAVA代码官方有直接示例，就比较简单，按如下方式即可：
1[root@ecs-acc1 ~]# cat SignDemo.java
2import java.io.UnsupportedEncodingException;
3import java.net.URLEncoder;
4import java.security.InvalidKeyException;
5import java.security.NoSuchAlgorithmException;
6import java.util.ArrayList;
7import java.util.Arrays;
8import java.util.Base64;
9import java.util.Collections;
10import java.util.HashMap;
11import java.util.List;
12import java.util.Locale;
13import java.util.Map;
14import java.util.TreeMap;
15import java.util.regex.Pattern;
16
17import javax.crypto.Mac;
18import javax.crypto.spec.SecretKeySpec;
19
20public class SignDemo {
21
22 private static final String SIGN_SEP = "\n";
23
24 private static final String OBS_PREFIX = "x-obs-";
25
26 private static final String DEFAULT_ENCODING = "UTF-8";
27
28 private static final List<String> SUB_RESOURCES = Collections.unmodifiableList(Arrays.asList(
29 "CDNNotifyConfiguration", "acl", "append", "attname", "backtosource", "cors", "customdomain", "delete",
30 "deletebucket", "directcoldaccess", "encryption", "inventory", "length", "lifecycle", "location", "logging",
31 "metadata", "mirrorBackToSource", "modify", "name", "notification", "obscompresspolicy",
32 "partNumber", "policy", "position", "quota","rename", "replication", "response-cache-control",
33 "response-content-disposition","response-content-encoding", "response-content-language", "response-content-type",
34 "response-expires","restore", "storageClass", "storagePolicy", "storageinfo", "tagging", "torrent", "truncate",
35 "uploadId", "uploads", "versionId", "versioning", "versions", "website", "x-image-process",
36 "x-image-save-bucket", "x-image-save-object", "x-obs-security-token", "object-lock", "retention"));
37
38 private String ak;
39
40 private String sk;
41
42 private boolean isBucketNameValid(String bucketName) {
43 if (bucketName == null || bucketName.length() > 63 || bucketName.length() < 3) {
44 return false;
45 }
46
47 if (!Pattern.matches("^[a-z0-9][a-z0-9.-]+$", bucketName)) {
48 return false;
49 }
50
51 if (Pattern.matches("(\\d{1,3}\\.){3}\\d{1,3}", bucketName)) {
52 return false;
53 }
54
55 String[] fragments = bucketName.split("\\.");
56 for (int i = 0; i < fragments.length; i++) {
57 if (Pattern.matches("^-.*", fragments[i]) || Pattern.matches(".*-$", fragments[i])
58 || Pattern.matches("^$", fragments[i])) {
59 return false;
60 }
61 }
62
63 return true;
64 }
65
66 public String encodeUrlString(String path) throws UnsupportedEncodingException {
67 return URLEncoder.encode(path, DEFAULT_ENCODING)
68 .replaceAll("\\+", "%20")
69 .replaceAll("\\*", "%2A")
70 .replaceAll("%7E", "~");
71 }
72
73 public String encodeObjectName(String objectName) throws UnsupportedEncodingException {
74 StringBuilder result = new StringBuilder();
75 String[] tokens = objectName.split("/");
76 for (int i = 0; i < tokens.length; i++) {
77 result.append(this.encodeUrlString(tokens[i]));
78 if (i < tokens.length - 1) {
79 result.append("/");
80 }
81 }
82 return result.toString();
83 }
84
85 private String join(List<?> items, String delimiter) {
86 StringBuilder sb = new StringBuilder();
87 for (int i = 0; i < items.size(); i++) {
88 String item = items.get(i).toString();
89 sb.append(item);
90 if (i < items.size() - 1) {
91 sb.append(delimiter);
92 }
93 }
94 return sb.toString();
95 }
96
97 private boolean isValid(String input) {
98 return input != null && !input.equals("");
99 }
100
101 public String hmacSha1(String input) throws NoSuchAlgorithmException, InvalidKeyException, UnsupportedEncodingException {
102 SecretKeySpec signingKey = new SecretKeySpec(this.sk.getBytes(DEFAULT_ENCODING), "HmacSHA1");
103 Mac mac = Mac.getInstance("HmacSHA1");
104 mac.init(signingKey);
105 return Base64.getEncoder().encodeToString(mac.doFinal(input.getBytes(DEFAULT_ENCODING)));
106 }
107
108 private String stringToSign(String httpMethod, Map<String, String[]> headers, Map<String, String> queries,
109 String bucketName, String objectName) throws Exception {
110 String contentMd5 = "";
111 String contentType = "";
112 String date = "";
113
114 TreeMap<String, String> canonicalizedHeaders = new TreeMap<String, String>();
115
116 String key;
117 List<String> temp = new ArrayList<String>();
118 for (Map.Entry<String, String[]> entry : headers.entrySet()) {
119 key = entry.getKey();
120 if (key == null || entry.getValue() == null || entry.getValue().length == 0) {
121 continue;
122 }
123
124 key = key.trim().toLowerCase(Locale.ENGLISH);
125 if (key.equals("content-md5")) {
126 contentMd5 = entry.getValue()[0];
127 continue;
128 }
129
130 if (key.equals("content-type")) {
131 contentType = entry.getValue()[0];
132 continue;
133 }
134
135 if (key.equals("date")) {
136 date = entry.getValue()[0];
137 continue;
138 }
139
140 if (key.startsWith(OBS_PREFIX)) {
141
142 for (String value : entry.getValue()) {
143 if (value != null) {
144 temp.add(value.trim());
145 }
146 }
147
148 canonicalizedHeaders.put(key, this.join(temp, ","));
149 temp.clear();
150 }
151 }
152
153 if (canonicalizedHeaders.containsKey("x-obs-date")) {
154 date = "";
155 }
156
157
158 // handle method/content-md5/content-type/date
159 StringBuilder stringToSign = new StringBuilder();
160 stringToSign.append(httpMethod).append(SIGN_SEP)
161 .append(contentMd5).append(SIGN_SEP)
162 .append(contentType).append(SIGN_SEP)
163 .append(date).append(SIGN_SEP);
164
165
166 // handle canonicalizedHeaders
167 for (Map.Entry<String, String> entry : canonicalizedHeaders.entrySet()) {
168 stringToSign.append(entry.getKey()).append(":").append(entry.getValue()).append(SIGN_SEP);
169 }
170
171
172 // handle CanonicalizedResource
173 stringToSign.append("/");
174 if (this.isValid(bucketName)) {
175 stringToSign.append(bucketName).append("/");
176 if (this.isValid(objectName)) {
177 stringToSign.append(this.encodeObjectName(objectName));
178 }
179 }
180
181 TreeMap<String, String> canonicalizedResource = new TreeMap<String, String>();
182 for (Map.Entry<String, String> entry : queries.entrySet()) {
183 key = entry.getKey();
184 if (key == null) {
185 continue;
186 }
187
188 if (SUB_RESOURCES.contains(key)) {
189 canonicalizedResource.put(key, entry.getValue());
190 }
191 }
192
193 if (canonicalizedResource.size() > 0) {
194 stringToSign.append("?");
195 for (Map.Entry<String, String> entry : canonicalizedResource.entrySet()) {
196 stringToSign.append(entry.getKey());
197 if (this.isValid(entry.getValue())) {
198 stringToSign.append("=").append(entry.getValue());
199 }
200 stringToSign.append("&");
201 }
202 stringToSign.deleteCharAt(stringToSign.length() - 1);
203 }
204 // System.out.println(String.format("StringToSign:%s%s", SIGN_SEP, stringToSign.toString()));
205
206 return stringToSign.toString();
207 }
208
209 public String querySignature(String httpMethod, Map<String, String[]> headers, Map<String, String> queries,
210 String bucketName, String objectName, long expires) throws Exception {
211 if (!isBucketNameValid(bucketName)) {
212 throw new IllegalArgumentException("the bucketName is illegal");
213 }
214 if (headers.containsKey("x-obs-date")) {
215 headers.put("x-obs-date", new String[]{String.valueOf(expires)});
216 } else {
217 headers.put("date", new String[]{String.valueOf(expires)});
218 }
219 //1. stringToSign
220 String stringToSign = this.stringToSign(httpMethod, headers, queries, bucketName, objectName);
221
222 //2. signature
223 return this.encodeUrlString(this.hmacSha1(stringToSign));
224 }
225
226 public String getURL(String endpoint, Map<String, String> queries,
227 String bucketName, String objectName, String signature, long expires) throws UnsupportedEncodingException {
228 StringBuilder URL = new StringBuilder();
229 URL.append("https://").append(bucketName).append(".").append(endpoint).append("/").
230 append(this.encodeObjectName(objectName)).append("?");
231 String key;
232 for (Map.Entry<String, String> entry : queries.entrySet()) {
233 key = entry.getKey();
234 if (key == null) {
235 continue;
236 }
237 if (SUB_RESOURCES.contains(key)) {
238 String value = entry.getValue();
239 URL.append(key);
240 if (value != null) {
241 URL.append("=").append(value).append("&");
242 } else {
243 URL.append("&");
244 }
245 }
246 }
247 URL.append("AccessKeyId=").append(this.ak).append("&Expires=").append(expires).
248 append("&Signature=").append(signature);
249 return URL.toString();
250 }
251
252 public static void main(String[] args) throws Exception {
253 SignDemo demo = new SignDemo();
254 demo.ak = "myak";
255 demo.sk = "mysk";
256 String endpoint = "obs.la-south-2.myhuaweicloud.com";
257
258 String bucketName = "ctslogstorage";
259 String objectName = "CloudTraces/la-south-2/2023/09/15/system/ECS/CloudTrace_la-south-2_2023-09-15T15-46-20Z_5bfdd257091735a3.json.gz";
260
261 // 若直接使用URL在浏览器地址栏中访问，无法带上头域，此处headers加入头域会导致签名不匹配，使用headers需要客户端处理
262 Map<String, String[]> headers = new HashMap<String, String[]>();
263 Map<String, String> queries = new HashMap<String, String>();
264
265 // 请求消息参数Expires，设置24小时后失效
266 long expires = (System.currentTimeMillis() + 86400000L) / 1000;
267 String signature = demo.querySignature("GET", headers, queries, bucketName, objectName, expires);
268 System.out.println(signature);
269 String URL = demo.getURL(endpoint, queries, bucketName, objectName, signature, expires);
270 System.out.println(URL);
271 }
272}
273
274[root@ecs-acc1 ~]# java SignDemo.java
275qAC0mnBMMRIsSMoaHZbMhhQQE64%3D
276https://ctslogstorage.obs.la-south-2.myhuaweicloud.com/CloudTraces/la-south-2/2023/09/15/system/ECS/CloudTrace_la-south-2_2023-09-15T15-46-20Z_5bfdd257091735a3.json.gz?AccessKeyId=myak&Expires=1695401956&Signature=qAC0mnBMMRIsSMoaHZbMhhQQE64%3D
四、最后
除了API方式，官方还提供了SDK和其他工具调用的方式，由于我这里后面要结合 Functiongraph 进行调用，所以选择了 API来实现。如果是平时主机上调用，建议直接使用官方给的 obsutil browser工具操作，更简单。
捐赠本站(Donate)
如您感觉文章有用，可扫码捐赠本站！(If the article useful, you can scan the QR code to donate))
- Author: shisekong
- Link: https://blog.361way.com/api-invoke-huaweicloud-obs/8988.html
- License: This work is under a 知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议. Kindly fulfill the requirements of the aforementioned License when adapting or creating a derivative of this work.