8 / 8
Dec 2016

NOTE REVISED DATE:



We are planning a major system maintenance event for late this Wednesday night / early Thursday morning (22Sep2016, formerly 21Sep) starting at midnight MST/PDT (GMT -7). Our system will be unresponsive until this operation concludes, most likely within half an hour, though as a very conservative outside estimate it could potentially last as late as 2am. Sporadic periods of unresponsiveness, though unlikely, may also occur afterward. Our support team will be on-duty during and after this period to test and verify success of the operation and respond to any user inquiries or bug reports that could result.



TECHNICAL SUMMARY:



For the past few months we have been preparing to migrate E-junkie services into the Amazon Web Services (AWS) server cloud, so this event will formally engage that transition for our live production system. This migration will eliminate technical and performance limitations of our legacy colocated hosting provisions and related hardware, bringing the top-notch performance, responsiveness, security and reliability of the Amazon cloud to benefit E-junkie's services for you, our valued clientele.



This migration necessarily entails a change of server IPs for E-junkie.com, which should not affect the vast majority of you; however, if you know that you have set up a custom post-checkout integration script that verifies the transaction-data submissions it receives by either checking the originating IP or doing a reverse DNS lookup to confirm the IP resolves back to e-junkie.com, those IPs will change frequently henceforth, so your script will now need to either: A) check the originating IP against the list of IPs you get from a forward DNS lookup of e-junkie.com, or B) eliminate IP/DNS verification in favor of the 'handshake' value we provide in our integration POSTs, as we document here:

1http://www.e-junkie.com/ej/help.integration.htm1



(If you didn't understand the paragraph above, it almost certainly doesn't pertain to you, so don't worry about it. :^)



We thank all of you in advance for your patience and understanding as we engage this major upgrade to E-junkie's infrastructure.

  • created

    Sep '16
  • last reply

    Dec '16
  • 7

    replies

  • 2.4k

    views

  • 2

    users

  • 4

    links

As the revised post above indicates, we are now tentatively rescheduling this maintenance event for Thu22Sep2016 Midnight-2am MST/PDT (GMT -7) due to unexpected issues we encountered with consistency of data between our legacy transaction log database and the new DB we'd be using on AWS, so we have rolled back to our legacy system until then.



This DB synchronization is the reason for the extended service outage required to finalize this migration; we need to stop all DB changes on our production system, then resync the legacy DB to the new one on AWS for however long that takes, then validate the data matches at both ends before we could open the migrated system on AWS to public access. This time, we found the data did not match, so we need to identify and resolve the reason for that mismatch before we can proceed.



Thanks again to everyone for your patience and understanding, with our apologies for the fruitless disruption in service this time; we promise it will all be worthwhile in the end. :^)

Our Transaction Log server will be unavailable today (Wed21Sep2016) starting from 2pm MST/PDT (GMT -7) in preparation for our rescheduled cut-over to AWS later tonight. This will give us ample time to resync our legacy log DB to the new one on AWS and validate data consistency without needing to take our whole system down, then when we perform the actual cut-over, we'll only need to resync new log entries that have been added since this afternoon, which should minimize the necessary downtime. Thanks for your patience and understanding, with our apologies for the temporary inconvenience.

We have decided to postpone finalizing our migration to AWS for one more day, so this event is now rescheduled for Fri23Sep2016 starting at midnight. Our log server will be back online momentarily, and service was not disrupted at all tonight due to some remaining ambiguities that need to be settled before we can proceed. Thanks again for your patience, with our apologies for any inconvenience this rescheduling may pose for you.

We have now completed the migration of E-junkie services into the Amazon cloud as explained above. While we have done our utmost before and after this morning's cut-over period to prepare, test, and verify that everything should continue to work the same as before, please do not hesitate to contact us if you notice any strange misbehavior of our site, shopping cart, Admin panel, or other services:

https://www.e-junkie.com/ej/contact.htm



PayPal should automatically attempt to resend any IPNs (Instant Payment Notifications) that failed to reach us during the downtime, but if you're seeing payments in PayPal that still aren't showing up in your E-junkie Transaction Log, see step C here for instructions to have PayPal resend any failed IPNs:

1http://www.e-junkie.com/ej/trouble.paypal.order-not-processed.htm1



Thanks again to all of you for your patience, cooperation and understanding while we engaged this major upgrade to E-junkie's infrastructure.

One hiccup we've encountered so far is that AWS will not allow us to send emails with a From: address @ another domain, so from now on we will be sending all thank-you emails as From: "[your Display Name here]" <client@e-junkie.com>, with your actual Display Email in the Reply-to: header instead, so your buyers can still simply reply to their thank-you emails to contact you.



This was eventually going to become inevitable even with our old system anyway, as email providers get more strict about this sort of thing to cut down on spam, phishing, and other such nuisance emails. Operations is looking into resending the handful of early thank-you emails affected by this hiccup right now, but you can also use "Reactivate Expired Links / Resend Email" in your Seller Admin to reissue them yourself.

2 months later

Hi - For a while now, it seems like many of my customers who have Comcast e-mail addresses might be getting silent blocking of E-Junkie download links. I don't know of a way to verify this, since I'm not getting bounce notices, but the incidence of Comcast customers not receiving download links seems to be disproportionately high.



To fix this, I was looking at getting listed on Comcast's feedback loop service. However, if I did this, it asks for a server IP address or CIDR. Looking at the headers for one of the latest E-Junkie download links, I see as you note that it comes from client@e-junkie.com, but since everything is now hosted on an Amazon SES server farm, I'm presuming the IP address is dynamic and not associated with any one server. It doesn't accept the IP address listed for our domain sprinter-rv.com, presumably because we're using CloudFlare for our CDN.



In the attached sample header, mx.google.com does give the download link a DKIM pass, resolving it as amazonses.com. It also gives it a pass on SPF with an accepted IP address of 54.240.8.64 (which resolves as a8-64.smtp-out.amazonses.com). I see that DKIM signing is in effect.



Anyway, the DKIM/SPF/DMARC stuff is a pretty deep rabbit hole, and it could be that this is an old issue that is actually solved now that E-Junkie has switched to this new Amazon SES setup. Could you just send a test E-Junkie download link going to a comcast.net e-mail address to the Amazon SES Mailbox Simulator, would that verify whether this is really a problem or not?



Thanks for any help!



Greg Keith

HiRes Media

www.sprinter-rv.com



============================





"Delivered-To: XXXXXXX@gmail.com

Received: by 10.37.66.193 with SMTP id p184csp1669559yba;

Mon, 5 Dec 2016 06:19:21 -0800 (PST)

X-Received: by 10.55.93.68 with SMTP id r65mr47415909qkb.84.1480947561084;

Mon, 05 Dec 2016 06:19:21 -0800 (PST)

Return-Path: <01000158cf5a7043-e6f01f95-09f5-4ca8-a5df-45213d518eb7-000000@smtp.e-junkie.com>

Received: from a8-64.smtp-out.amazonses.com (a8-64.smtp-out.amazonses.com. [54.240.8.64])

by mx.google.com with ESMTPS id v75si8965001qkl.266.2016.12.05.06.19.20

for <XXXXX@gmail.com>

(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);

Mon, 05 Dec 2016 06:19:21 -0800 (PST)

Received-SPF: pass (google.com: domain of 01000158cf5a7043-e6f01f95-09f5-4ca8-a5df-45213d518eb7-000000@smtp.e-junkie.com designates 54.240.8.64 as permitted sender) client-ip=54.240.8.64;

Authentication-Results: mx.google.com;

dkim=pass header.i=@amazonses.com;

spf=pass (google.com: domain of 01000158cf5a7043-e6f01f95-09f5-4ca8-a5df-45213d518eb7-000000@smtp.e-junkie.com designates 54.240.8.64 as permitted sender) smtp.mailfrom=01000158cf5a7043-e6f01f95-09f5-4ca8-a5df-45213d518eb7-000000@smtp.e-junkie.com

DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw; d=amazonses.com; t=1480947560; h=To:Subject:Sender:From:Reply-To:Content-type:Content-transfer-encoding:MIME-version:Message-Id:Date:Feedback-ID; bh=feTnm7jAMOkWRYjGRaeL9qJMKNN7vGdg/z7cgTn5/qs=; b=uM2QCqLQrEMMrmStEVnOAzc/TDpCVX7M0aEtTneQiYegQUdLEjzubGUQc8ZOQ8NQ XYlVc1/sDoWLG1jqqyfLzw+PHHuHGTZTTRXoJxq0+atEuVkO8q7ktP+K8Qefp3L1wZ8 SUxzKFWHwhS9km+6cRWy8+syAPbKceBlj2s0yPo0=

To: XXXXXX@hotmail.com

Subject: Sprinter RV Conversion Sourcebook Purchase

Sender: <client@e-junkie.com>

From: HiRes Media LLC <client@e-junkie.com>

Reply-To: HiRes Media LLC <XXXXX@sprinter-rv.com>

Content-type: text/plain; charset=UTF-8

Content-transfer-encoding: 8bit

MIME-version: 1.0

X-Priority: 3

X-Mailer: E-junkie.com

Message-ID: <01000158cf5a7043-e6f01f95-09f5-4ca8-a5df-45213d518eb7-000000@email.amazonses.com>

Date: Mon, 5 Dec 2016 14:19:20 +0000

X-SES-Outgoing: 2016.12.05-54.240.8.64

Feedback-ID: 1.us-east-1.nDFS600l1LKSkMROJcwS7qwlskwpwZiZPs47zfiAJsg=:AmazonSES"

Most likely I would suspect the issue with email to Comcast is simply a matter of spam filtering and/or delivery delays at their end. We process orders, including sending thank-you emails, immediately when we get confirmation of completed payment, but once the receiving mailserver accepts the message, ultimate delivery to the recipient's actual inbox is beyond our control or insight. Delivery delays of up to an hour or more to major email providers are not uncommon, and this may affect some messages but not others depending on which inbound mailserver each message happens to hit.



That said, I'll refer this to Operations for further investigation. Have you been seeing this issue more or less often since Sept. 23rd, when we finalized migration of our services into the Amazon cloud? Since then, your Transaction Log shows about 3% of your orders have been from buyers @comcast.net; of those Comcast buyers, about 85% were able to complete their download in only 1 or 2 Attempts, and I'm seeing only one Comcast buyer who did not make any Attempts at all on their original link, though you did send them a free replacement link, on which they made 1 Attempt.



I would expect that Amazon SES is already registered for the Comcast Feedback Loop, which should cover that angle as the messages are actually being sent by their servers rather than yours or ours, and they would forward us any feedback regarding mail sent as From: our domain. Note this only pertains to email the recipient deliberately marks as spam, which is unlikely for emails delivering a link they'd paid for, and this feedback is really only used to keep complaint rates down by seeing which kinds of message are most likely to get marked as spam. Speaking of, for this reason and due to automated spam filtering, we recommend keeping your thank-you emails as brief and succinct as possible, avoiding unnecessary link URLs and promotional language that could trigger spam filters or a user's perception of the message as spam.



The Amazon SES Mail Simulator would not be useful to test messages to comcast.net; it only provides specific test addresses @simulator.amazonses.com that messages can be sent to, each of which respond to incoming messages in a predetermined way (e.g., accepts the message, or bounces it as undeliverable, or autoresponds, or marks it as spam, etc.), so the sender can see what happens in each of those scenarios.