Fixed an error when placing objects in a different bucket than the region running the Lambda function

[handlename]

If the region running Lambda function is not same to destination S3 bucket, AWS API returns errors like berow:

api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

To avoid this, added codes to init AWS API Client for each destination.

before (v0.1.1)

$ LOG_LEVEL=debug s3-object-router \
    -bucket s3://***-cloudfront-logs/***/E12***********.2024-05-16-01.137420f5.gz \
    -parser cloudfront
    -time-parse true \
    -time-key datetime \
    -time-format 2006-01-02T15:04:05Z \
    -key-prefix cloudfront/{{ .x_host_header }}/dt={{ .datetime.Format "2006-01-02" }}
2024/05/17 14:12:34 [debug] option: router.Option{Bucket:"***-test", KeyPrefix:"cloudfront/{{ .x_host_header }}/dt={{ .datetime.Format \"2006-01-02\" }}", TimeParse:true, TimeKey:"datetime", TimeFormat:"2006-01-02T15:04:05Z", LocalTime:false, TimeZone:"", Gzip:true, Replacer:"", Parser:"cloudfront", PutS3:true, ObjectFormat:"none", KeepOriginalName:false, replacer:router.replacer(nil), recordParser:router.recordParser(nil), newEncoder:(func() router.encoder)(nil), newBuffer:(func() router.buffer)(nil), timeParser:router.timeParser{layout:"", loc:(*time.Location)(nil)}}
2024/05/17 14:12:34 [info] run s3://***-cloudfront-logs/api/E12***********.2024-05-16-01.137420f5.gz
2024/05/17 14:12:35 [info] route s3://***-test/cloudfront/***.***.jp/dt=2024-05-16/cd01418c50ac4619208a99ab531dede4deabfb4cf5aea9757209003c359d12c5.gz 631 bytes
2024/05/17 14:12:35 [info] starting put to s3://***-test/cloudfront/***.***.jp/dt=2024-05-16/cd01418c50ac4619208a99ab531dede4deabfb4cf5aea9757209003c359d12c5.gz
2024/05/17 14:12:35 [error] operation error S3: PutObject, https response error StatusCode: 301, RequestID: C3D0037FHHA3JYDF, HostID: Me95q2zrA3tGaFxCSmg+W4M8BEuG+H/5OiPk3DV8Ifgg2jjnF5phL6DED9gO8Ni2WJ+JBkDnBQ0=, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
make: *** [run] Error 1

after

$ LOG_LEVEL=debug go run cmd/s3-object-router/main.go \
    -bucket s3://***-cloudfront-logs/***/E12***********.2024-05-16-01.137420f5.gz \
    -parser cloudfront
    -time-parse true \
    -time-key datetime \
    -time-format 2006-01-02T15:04:05Z \
    -key-prefix cloudfront/{{ .x_host_header }}/dt={{ .datetime.Format "2006-01-02" }}
2024/05/17 14:12:48 [debug] option: router.Option{Bucket:"***-test", KeyPrefix:"cloudfront/{{ .x_host_header }}/dt={{ .datetime.Format \"2006-01-02\" }}", TimeParse:true, TimeKey:"datetime", TimeFormat:"2006-01-02T15:04:05Z", LocalTime:false, TimeZone:"", Gzip:true, Replacer:"", Parser:"cloudfront", PutS3:true, ObjectFormat:"none", KeepOriginalName:false, replacer:router.replacer(nil), recordParser:router.recordParser(nil), newEncoder:(func() router.encoder)(nil), newBuffer:(func() router.buffer)(nil), timeParser:router.timeParser{layout:"", loc:(*time.Location)(nil)}}
2024/05/17 14:12:48 [info] run s3://***-cloudfront-logs/api/E12***********.2024-05-16-01.137420f5.gz
2024/05/17 14:12:48 [debug] bucket region for ***-cloudfront-logs is ap-northeast-1, added to cache
2024/05/17 14:12:49 [info] route s3://***-test/cloudfront/***.***.jp/dt=2024-05-16/cd01418c50ac4619208a99ab531dede4deabfb4cf5aea9757209003c359d12c5.gz 631 bytes
2024/05/17 14:12:49 [debug] bucket region for ***-test is us-east-1, added to cache
2024/05/17 14:12:49 [info] starting put to s3://***-test/cloudfront/***.***.jp/dt=2024-05-16/cd01418c50ac4619208a99ab531dede4deabfb4cf5aea9757209003c359d12c5.gz
2024/05/17 14:12:49 [info] completed put to s3://***-test/cloudfront/***.***.jp/dt=2024-05-16/cd01418c50ac4619208a99ab531dede4deabfb4cf5aea9757209003c359d12c5.gz

[fujiwara]

putToS3 might be called from multiple goroutines in parallel, so we have to guard concurrent map access. Please consider using sync.Mutex or another lock mechanism.

[fujiwara]

This is also map access to read, which should be guarded by a mutex.

[handlename]

Fixed.
Apologies for sloppy commit 🙇

[fujiwara]

👍