latency impact of distance between client to lambda VS lambda to dynamodb table
Question
If a service’s DynamoDb (DDB) is in a region from some users, is it better to call DDB from a lambda close to the end user OR close to the DDB?
(ie: if DDB is in us-east-1, but some users are in us-west-2, does it make sense to deploy a lambda in us-west-2 OR should they just call back to us-east-1)
Note
This assumes that DDB Global Tables are not viable (ie: already have a global table in us-west-2, it probably would not make sense to replicate it into us-west-1).
Test
3 setups, for a ddb in us-east-1
- Lambda in us-west-2
- Lambda in us-east-1
- Cloudfront infront of us-east-1 Lambda (for edge routing)
Additionally a base line was generated by running the same test against a us-west-2 Lambda talking to a us-west-2 DDB.
The Lambda was a single js file running on nodejs-16.x with 125MB of memory and no provisioned capacity. The Lambda would perform a single GetItem request for the ‘350KB’ row in the table and return the result. This item was just an ID and a 350KB string attribute so as to generate some network transfer latency.
Process
From an ec2 in us-west-2, a 5 min load test with 30 concurrent users was run using artillery against each Lambda url / Cloudfront distribution.
Results
BASELINE
http.codes.200: ................................................................ 859
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 859
http.response_time:
min: ......................................................................... 108
max: ......................................................................... 1836
median: ...................................................................... 202.4
p95: ......................................................................... 528.6
p99: ......................................................................... 620.3
CROSS REGION DDB
http.codes.200: ................................................................ 914
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 914
http.response_time:
min: ......................................................................... 526
max: ......................................................................... 3446
median: ...................................................................... 1652.8
p95: ......................................................................... 1978.7
p99: ......................................................................... 2186.8
CROSS REGION LAMBDA
http.codes.200: ................................................................ 914
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 914
http.response_time:
min: ......................................................................... 167
max: ......................................................................... 1936
median: ...................................................................... 262.5
p95: ......................................................................... 596
p99: ......................................................................... 645.6
CLOUDFRONT INFRONT OF CROSS REGION LAMBDA
http.codes.200: ................................................................ 882
http.request_rate: ............................................................. 3/sec
http.requests: ................................................................. 882
http.response_time:
min: ......................................................................... 202
max: ......................................................................... 1859
median: ...................................................................... 399.5
p95: ......................................................................... 788.5
p99: ......................................................................... 925.4
Did running the same test from my home computer to see if starting outside of the AWS network would have any affect?
it did not
Conclusion
Best to just call out to far away region directly (it had the closest latencies to the base line for all the stats)
Notes
- repo with infra + lambda + test config is available at https://github.com/BASED-EDGE/cross-region-latency
- this only covers a limited use case, best to investigate for yourself
Further questions
- What about different services from just than just DDB (ie: RDS, S3, SNS)?
- Does the same thing apply to DDB write requests? YES
- What effect does using different regions have on the above results (ie: ca-central-1 to ap-southeast-2)
- Why does remote DDB perform so much worse than remote lambda? the lambda is not transforming the data (apart from
JSON.stringify()
) so the network latency would be assumed to be very similar. - How would a cross region AWS PrivateLink endpoint change the results?