I noticed a few weeks ago that my kitchen Echo Show was constantly uploading data at about 1.5Mbps, which is not insignificant. I disconnected it from the wifi, and when reconnected it seemed to stop uploading and I thought nothing more of it. Last night I noticed it again, and that it had uploaded over 40 gigabytes since last connecting.
I did what any sensible person would do, I tweeted / tooted about it and then went to bed. In the morning, surprise surprise, nothing had changed, it had uploaded over 10 gigabytes overnight and no one had replied with any useful ideas.
So it was time to do some digging myself. I have SSH enabled on all my Unifi equipment so a quick SSH session in the AP it's connected to and we can run tcpdump to see what's going on:
APOffice-BZ.6.5.26# tcpdump host 192.168.1.139
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:05:22.288182 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], ack 18886, win 1369, options [nop,nop,TS val 218210814 ecr 3874437661], length 0
11:05:22.361977 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [P.], seq 5368261:5368863, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 602
11:05:22.364317 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5368863:5370311, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.364360 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5370311:5371759, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.364384 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5371759:5373207, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.364406 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5373207:5374655, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.364429 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5374655:5376103, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.364453 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5376103:5377551, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.364476 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5377551:5378999, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.366126 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5384791:5386239, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.366213 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5386239:5387687, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.366242 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5387687:5389135, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.366266 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5389135:5390583, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.366289 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5390583:5392031, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.366310 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5392031:5393479, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.372187 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5397823:5399271, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.372229 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5399271:5400719, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.372253 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5400719:5402167, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.372278 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5402167:5403615, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.372303 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5403615:5405063, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.372327 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5405063:5406511, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.372350 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5406511:5407959, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.374126 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5416647:5418095, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.374155 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5418095:5419543, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.374178 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5419543:5420991, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.374202 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5420991:5422439, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.374225 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5422439:5423887, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448
11:05:22.374248 IP 192.168.1.139.50388 > ec2-54-209-44-8.compute-1.amazonaws.com.443: Flags [.], seq 5423887:5425335, ack 18886, win 1369, options [nop,nop,TS val 218210821 ecr 3874437661], length 1448That's a lot of big TLS packets heading out to an EC2 machine, with no reverse DNS entries. Let's dump all the raw packets from the Echo to a file, and then see what Wireshark can make of it:
ssh [email protected] 'tcpdump host 192.168.1.139 -w -' > ~/Downloads/echo-traffic.pcapI left that running for a 'few minutes' and ended up with about 200 megabytes of raw packets, loaded them into Wireshark and confirmed that it was almost all entirely TLS encrypted traffic. But you can't transfer data over a TLS encrypted connection without first negotiating that connection so let's filter for TLS handshakes ssl.handshake.type >= 1 and see what hostname the client (the Echo) is presenting:
Well look at that, there are a lot of connections to device-metrics-us.amazon.com which seems to have quite a few IP addresses. Let's throw all those in a filter and see how much of the traffic is heading there:
Looks like, pretty much, all of it is heading that way.
So how do I stop this, well I could contact Amazon support, as suggested, but I don't fancy explaining tcpdump & Wireshark to some front-line support representative. Instead, I'll just add device-metrics-us.amazon.com to my DNS blocklist and be done with it. No more excessive outbound traffic.
If anyone from Amazon has read this far and can do anything on their side to investigate further, do get in touch. My Twitter DMs are open, or you can probably find my email address, phone number, home address, childrens' names, favourite foods etc in those device metrics.