18 / 22
Feb 2024

All PayPal IPN failed still. the status showing Retrying for all of them.

The ipn_res_cart.php IPN Notification URL is what we pass to PayPal for every checkout, which overrides any manual IPN settings in the seller's PayPal account, so that's fine. We only suggest using the o_plug.php URL to re-enable IPN after PayPal completely disables IPN for a seller's PayPal account, since that requires manually entering something as a new Notification URL.

Things may still be sporadically unstable/erratic for a bit yet, as our engineers continue their recovery work to restore full functionality, so that's why PayPal may be having trouble reaching us with some IPNs still.

Thanks! And, buy now button start to work, but, add to cart and view cart button are not working, not sure is related, or it is my own problem.

Not your problem; our engineers are still working on getting things fully functional and stable again, so please be patient with us.

Still problems? I've been trying to upload product images for a couple of hours now.

Not working. Getting errors all the time now. Incredibly unstable at the moment.

I received a message from PayPal about the purchase, stating that the purchase price has been received, but there is no trace of it on the webshop (Ejunkie). The webshop hasn't sent the product to my buyer, and there's no indication anywhere that any purchase has been made.

I'm having the same issue. I've found at least two PayPal purchases that e-junkie did not send a download link. No record of them at all in my e-junkie logs and no email from e-junkie explaining why the customer was NOT sent a download link. This is really bad because some transactions are generating download links and others are not. It's very hard to sort through PayPal emails and try comparing them to e-junkie logs to find out what's missing.

What's the solution here? Is e-junkie going to eventually get these customers their download links or not?

I've now found five PayPal purchases for which e-junkie did not send download links. In fact, I've only found one paid purchase where the customer received a download link. Free downloads are getting their download links just fine it seems, but HUGE problem getting PayPal purchases to generate download links from e-junkie. Things are definitely not working yet. After customers pay with PayPal, e-junkie doesn't sent a link. Bad situation.

Now I just had a purchase where customer made a purchase of free and paid items and appears to have received the download links for all of them. So, whatever is going on appears inconsistent.

Will need guidance on how to deal with customers who completed PayPal purchases but did not receive download links. Will we have to manually generate them and send to customers or will e-junkie eventually send them?

Just finished going through all my PayPal purchases from yesterday and today, and it looks like they have now all been sent download links now.

Thanks to everyone for your patience and understanding while we got things back in check, with our apologies for the extended trouble. Now that we have things stabilized again, here's what we now know about the cause of the outage:

Apparently this was all a result of someone trying to extort money from E-junkie's owner. They'd claimed they devised some obscure hack that could take down our system, which they'd reveal if they were paid a hefty ransom, then launched this attack as a purported "demonstration" of their exploit. However, our DevOps engineers soon found the attack wasn't anything like the obscure hack they'd alluded to, so that was evidently a bluff, and they were hoping we'd just pay the ransom before we discovered their ruse.

Rather, all signs indicated their "demonstration" was nothing other than an ordinary DDoS (Distributed Denial of Service) attack flooding our system with bogus connection attempts, which is rather like flash-mobbing a physical store with rabble-rousing loiterers, so legitimate buyers can't get in to shop. These typically use a "bot net" comprising multitudes of computers scattered worldwide (typically malware-infected PCs) to make every hit come from a different IP address, making it ineffective to block any particular IP. Our DevOps team spent the afternoon and evening manually fending off the bogus traffic as best they could while enabling and testing various automated countermeasures to curb the attack without such manual intervention, in a way that would not also adversely affect our services in other ways, so that involved a bit of trial and error to arrive at an effective solution.

While all that bogus DDoS traffic was slamming our system, some PayPal IPNs (Instant Payment Notifications) may not have reached us. We depend on receiving IPN to know when a buyer's PayPal payment is Completed, so we can then process their order accordingly. Fortunately, PayPal does automatically retry Failed IPNs a few more times, so many of those may have come through by now.

If you're still seeing any Completed payments in PayPal that aren't listed in your E-junkie Transaction Log, that indicates we did not process that order yet, in this case most likely because the IPN for that payment couldn't get through to us. You can check for any such Failed IPNs and have PayPal retry them, at which point we should then be able to process that buyer's order as usual. The troubleshooting steps on this help page starting from Section C, Step 2 will walk you through how to do that:

Thank you for your transparency. It is very much appreciated.

Thanks so much for letting us know what happened and what to do if we had any trouble. Thanks, too, to the DevOps engineers who worked so hard to minimize problems, find the source, and stop it.

1 month later

:musical_note: Guess who's back, back again? :musical_note:

Looks like our system is facing another DDoS attack, same attack pattern as a month ago.

Our DevOps engineers are aware of it and still have our countermeasures enabled from last time, which appear to be working as intended, but there may be brief and sporadic periods when our system may be slow or unable to respond until the attack waves subside.

Thanks to everyone for your patience and understanding while we fend this off and ride out the storm.

Looks like things have been stable for a few hours now, so this issue should now be resolved. Thanks again for your patience and understanding, everyone, with our apologies for the brief disruption.