/biz/ - Business & Finance

>>7478580
brainless fudder here
AHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHHAHAHAHAHAHAHAHAHAHAHAHAHAHAHH exit scammed

I worked for a major bank as a big data expert and with a large scale hadoop file system. Transferring data between clusters can take a long ass time, but they should really have had a hot backup ready to go. Not sure what the issue is unless theyve had some hardware failure.

CHINKED

WHY DID I NOT FUCKING LISTEN WHY DID I LEAVE MY SHIT ON THIS CHINK EXCHANGE AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

>>7478624
I feel like this explanatory tweet is just going to confuse and piss off people

looks like theyre doing a really good job at keeping people updated

>>7478606
dude has a net worth over $2b, even after the crash

sure he'll become a fugitive and live undercover for the rest of his life so he can get a few more b

prove this fudder wrong!

You gonna be drive that Lamborghini soon, right?

>>7478637
The problem is theyre being pretty vague with what the exact issue was (which is reasonable), so people with experience in the field cant really gauge if we should be worried or not.

seems like the type of shit that would happen every fucking week back when i played world of warcraft and their maintenance took twice as long as it should have

>>7478683
fugitive? from who? the world police? crypto isnt regulated u mong. also he doesnt actually have 2 bil, hes going to have to slowly cash out

Programmer professionally since 07. Yes, these kinda things happen. Everything should be fine. Besides, you have your coins in your own wallets, right?

>>7478774
no (sane) person in the world has over a billion in cash, all billionaires would need to 'slowly cash out' if they wanted to convert all their net-worth to dollars

>>7478580
I'm a software developer.

It's really hard to know. It depends how the data is stored, etc.

There are so many variables, such as do they need to re-process data from a data warehouse to get it back into their processing system or is it literally just loading backups.

It's not unheard of for backup recovery to take this long, but they should have had a hot failover.

I'm still not sure what exactly happened, but I'd give it a day or two before I'd freak out.

I mean, they are probably freaking out over there, and started recovering from a backup, thought they could do it a quicker way and cancelled the original recovery, realized the second way doesn't work, and had to start over.

They also may need to fix infrastructure before they can even reload/process data, and that could take days depending on their deployment environment.

>>7478821
yea i know that. thats what i meant, a crypto billionaire doesnt mean shit. if anything he'll just pull an excuse like "oh we were hacked" so he wont look like the bad guy. scapegoat

I'm a lead distributed systems engineer at a top 100 internet company involved in AdTech. Having an outtage this major should be VERY rare, but given that it has happened and that they seemingly had no disaster recovery plan, it sounds normal. People in another thread mentioned the tweeting by the CEO sounds nervous, and it should be, because every dev during an outtage like this is shitting himself and probably pulling all nighters trying to fix the issue.


Also, blockchain is garbage. I know a lot of you probably never have ran your own blockchain node (aka wallet) and probably use exchanges, but you have to download the full blockchain (or a large % of it) before you can do anything. Mist has a known bug which prevents a majority of new users from synching the entire network, and thats what I'm guessing is causing the delay.

>>7478774
>fugitive? from who? the world police? crypto isnt regulated u mong.

you're a retard. all inernational law enforcement agencies including the FBI would be after him if he left with people's property. binance is even fully legal to use for americans

>>7478580
my dad works for binance, he said meet him in Bahamas for a looooong vacation?

>>7478580
No reason to panic when you can just check their address.

https://etherscan.io/address/0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be

>>7478580
Software engineer here. Databases can be very unwieldy beasts, especially one managing the transactions for something with as much SCALE as the #1 crypto exchange.

I'm pretty confident that things will be restored 100% accurately with minimal hiccups though. His team aren't amateurs.

>>7478775
>Besides, you have your coins in your own wallets, right?

>>7479028
and should he get caught how exactly will he get prosecuted? under which countries laws? ultimately it was you who decided to take the risk

question: if binance has shown that their data is fucked/fuckable, how can their records possibly be used by the IRS to enforce taxes? seems to me like iffy information means an easy legal battle for the traders

this is a fking joke.

>>7479017
>I know a lot of you probably never have ran your own blockchain node (aka wallet) and probably use exchanges, but you have to download the full blockchain (or a large % of it) before you can do anything.
This is what annoys me about this board. I've never heard of transaction fees until people were too lazy to download 20+ GB of data for owning their own wallet

I signed in 7 hours ago and my shitcoins were right where I left them. Let the FUD spread, I like cheap ETH

>>7479135
Transaction fees are baked into the protocol. You have to pay them even if you use your own wallet, although some blockchain protocols allow you to adjust the fee depending on how urgent you need your transaction verified.

>>7478580

Coder here.

Ummmm.....yes? Shit breaks?

Honestly though, I've noticed a lot of small UI glitches and general messiness on Binance. The problem is that testing can never demonstrate the absence of bugs, just their presence. Which is less usel than it sounds.

Meaning whatever pack of of pajeet built that shit undoubtedly fucked up somewhere. Gg Binance.

>>7478580
software developer (and part time server admin) here
they clearly didn't have a procedure in place on how to restore backed up data
if they did, they wouldn't be in this position having to try multiple methods to see what works best
you should already know that ahead of time
preferably you'd have a one-click solution to pull the backups from somewhere and run a restore script

>>7478580

My personal opinion.... after the accident they realized that their backup was fucked, and that they've lost their data.

They don't want the public to know about this, so they are desperately fighting to recover the data, or are trying t manually rebuild it based on the public ledger.

>>7478624

My guess is that they were depending on AWS auto-scaling (as it should have) but failed. This has happened for many Amazon customers.

>>7478580
Ive never configured a replica cluster, but if theyre literally restoring an image of master or a backup of the master, it could easily take DAYS if the thing is 100+TB. Even moving that data is slow, let alone writing it and rebuilding indexes etc.

Is this a good time to buy bnb?

>binance didn't backup their privkeys

>>7479265
Yeah they said it was a server issue which makes it much less likely that the data is actually corrupt and there was some service/hardware failure like you mention.

>>7479256
>>7479256

Why would he keep talking if this was an exit scam?

Also, hes already a billionaire.

Don't be stupid.

>>7479317
Hardware/infrastructure issues often result in data corruption. So it could be both, or neither

HEHEHEY
CARLOS MATOS HERE
BINANCEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

>>7478580
Call me a fudder, but not in love with the idea that they decided to not go for method 3? I don't know shit about how this works but I'm assuming there's a good reason why it's slower.

>>7479361
may not be an exit scam but there might be a pretty big problem where you may never see your money again

>>7479378

AWS has a live triplicate policy (unless they have changed), but the backups are not stored on hot devices...sort of explains why it would take this long if primary failed.

Insider here. All three "methods" would fail. Sell everything you can now. You have 8 hours

I've actually built a system that could fail in a similar way way. There was a master database that was doing replication streaming to a bunch of other nodes that had a copy of all its data.

https://wiki.postgresql.org/wiki/Streaming_Replication

if the master fails, we'd need to rebuild the entire cluster and resync the data. but we had a redundant master so if it failed, we could switch to the other one. never happened, thankfully.

I wonder why they arent letting the sync finish. If it take 20+ hours to sync, I'd say let it run and just wait. I cant think of any faster options than that. I'm not sure how they can take something that should take 9 hours and make it take 1-2 hours. That doesnt make any sense to me.

>>7479028
yeah, just like how the fbi is all over Bittrex for stealing literally tens of millions from a small selection of about 10,000 accounts.

>>7479485
depends on the data
maybe they can split it up into 4 parts and run 4 smaller backups to 4 different databases at the same time, making it 4x faster

>>7479478
Method 1 is turning it off them back on again.
Method 2 is is finding where the fuck they put the HDD with the backup.
Method 3 is having CZ recreate the entire checksum on pen and paper via his photographic memory.

>>7479543
kek

>>7478624
Asian SWE from big data vendor. Obv I don't know what happened at Binance, but their explanation so far is not inconsistent with the problems you do have operating these systems in the real world.

Imagine Binance loses all their private keys and all the funds inside are permanently locked. Significant deflation + USDT getting wiped off the books. Could be awesome for the market as a whole.

>>7479864

Not to mention...

.. destroys confidence in all the other exchanges.
... used as excuse to regulate the entire industry.

>>7479900
>DEVOPS199 JUST DESTROYED THE MARKET SELL SELL SELL
In the long term the deflation is way, way more relevant

Yes, it’s totally normal. This is the largest exchange in existence. I guarantee their dB is in the high terabytes, and takes time to restore. Also, you can see their wallet addresses, which if you remember, is the whole point of crypto. Their coins haven’t gone anywhere

>>7480017
Not our problem, we pay or fees without delay and only ask for service.
Keep the fucking site online or stay out of business.
Fucking binance shills.

>>7480061
I’m not saying it’s not shitty business practice, but everyone saying they were hacked is retarded

>>7480061
Then don't use binance asshole

>>7478580
I'm an actual Technical support guy for enterprises/businesses.

Transferring terrabytes of data takes hours, if not days. Real things man.

>>7480108
Our HDFS is so big once when we had a failure we had to fly backup HDs from cold storage to our datacenter

>>7480152
Most newfags won’t remember this but one time 4chan was down for a couple days because moot had to drive the backup discs from Texas to CA when he changed service providers.

If the post I saw earlier about them giving free ETH as compensation is true, I bet it comes at the cost of agreeing not to sue.

Their exchange, down for over a day. People can't get to their money during one of the most critical stock market days in years.

Quite the fuck up.

You expect more out of an unregulated industry where anonymous people trade a commodity with an indeterminate value.

>all these posts popping up suddenly
>developer here, everything will be just fine!

>>7480313

Based on what?

Unless you have access to their equipment, you're really in no position to know how serious the problem is

>>7480265
>People can't get to their money during one of the most critical stock market days in years.
See now you have to wonder. Is it really a coincidence? Right after the SEC shilled crypto?

>>7480313
my dad works at microsoft

This shit bout to get ugly niggas, once the binance paid shills come to defend their master means something is going up.

>>7480313
Do you think they could try to pose as hedge fund managers when they want to dump their bags?

Worked for APX, Sony and the UN as a software developer. This is a typical procedure just transfering data. Takes hours or days. It will be OK.

>>7480351

I'm still wondering about my original theory, which is this...

... they were trading on their own account. Borrowing customer coin to long or short the various currencies, and they got caught upside down with the price action earlier in the week.

More than the correlation to the stock market, I find it suspicious that this happened right after the BTC price went crazy with the below 6000 up above 7400 price action we saw for that one hour.

>>7480265
> If the post I saw earlier about them giving free ETH as compensation is true
Where did you see this?

i thought they are doing a system upgrade, why are they doing data recovery?

>>7478580
this is the dumbest shit I've ever seen. If option 1 is the fastest and he lists no downsides..why wouldn't they just use that option and not tweet about 2 other much longer options?

fucking jokers

>>7479864
That didn't happen here because Binance moved 30k BTC from their cold wallet to their hot wallet at 7pm.

>>7480061
shit happens
disclaimer: I only have ~20 BNB in Binance
I converted from USDT -> USD during the Tether FUD

>>7480265
> If the post I saw earlier about them giving free ETH as compensation is true,

This fucking guy

>>7478580
the real question is why arent these noobs using VEEAM

>>7479161
right? I literally logged in 10 times today, nothing is missing. only around 5PM EST it was a little slow but I was still able to get in

>>7478649

time for binance refugees to join

THE PEOPLES EXCHANGE

ku coin

>>7478580
networking specialist here. looks like someone fucked up majorly and probably taking ages to initiate their disaster recovery

>>7479076

So they have some BTC

Doesn't mean they'll give you any

LMAO

>>7479161
yeah, same. my shit is fine

still, going to move everything to a wallet. this is a good wake up call

>>7481233

In Chinese, "disaster recovery" translates to....

... move to the hills, never seen again, sorry about all that money you used to have, but I need it more for hookers and blow.

>>7481132

I use veeam

>>7479106
What if the IRS is behind all of this?
In order to get a record of everything.

>>7479017
Binance has infinate money. They should have their entire cluster synced to multiple instances at multiple DCs.
Infinite money = NEVER go down unless youre retarded or stingy on hosting.

There is definitely a major fuckup at play here. More than they let on for sure

"Cheng, here's the money. We'll dump on them when they least expect it. Now the exchange back to working order or else.

>>7479256
>>7479342

i remember a tweet of him saying one of his engs threw up when they saw how long it would take(and why i assume)

maybe he saw the data was donezo and then i could understand that reaction.

>>7481432
Do you know how hard it is to keep multiple clusters in sync with the transaction volume they have? Its VERY fucking hard. I had to build something at work that did something similar. What do you think the latency SLA is for their backups? Daily backups? Hourly? Lol if you think they use replicated writes cross-DC at their transaction volume

>>7481797
>Lol if you think they use replicated writes cross-DC at their transaction volume
Its definitely possible. The DCs would be in same region and have massive bandwidth between them.
Throwing money at it makes it possible.

>>7481432
They did, but the problem is supposedly that the sync to the backup cluster got fucked, so they had to offline the master db in order to do a full resync. Their database size could easily be in the multi-petabyte range. That said, their architecture should allow for multiple backup clusters, but most likely it's some bullshit like a single mysql slave or some shit that they've had in place since before they got big.

>>7481846
You do not know what you are talking about. Atomic writes cross-DC is not feasible at their scale due to noise on the write. You have to make writes atomic across all data centers which means your write has to block until all data centers confirm ack the write.. If any fail the write must be retransmitted. I'm not even going to dignify any of your noobish comments anymore.

Okay my jimmies are starting to get pretty rustled here what the fuck

>>7481899
of course you could batch it. It doesnt have to be synced 100% in real time.
Its enough to be able to fall back onto a 5-10 min old version in case of catastrophic failure of your main production system.

eating the loss on compensating users that missed out in those 10 minutes is smaller than going down for hours on end.

If you think with unlimited budget you cant create a setup thats almost impossible to down, you are a noob adminlet.

>>7481869
>only having one backup cluster

Who would GOX now and not 1 month ago?

>>7481899
What noise you fucking retard. Its very feasible

guys why didnt they use a distrusted ledger for this

so obvious

>>7482265
have you followed the gox court case? If you lose most of your user's coins, after courts, you will only have to pay the fiat value at the time you lost them.

Mt Gox filed for bankruptcy because they were insolvent. However, the coins they have now became worth several times their pre-hack networth. He gets to keep the difference.

If binance was going to gox (which nip law now has a precedent for) they wouldn't do it at ATH, they would wait for a big dip (which is a LOT easier to predict with access to unfiltered market data).

>>7482383
Shit you are right, I forgot about that loophole/precedent.

Well gentlemen and traps, it's been a pleasure to fud and shill with you, see you in the valhalla or wherever you go after going hero.

> In heaven all the candlesticks are green

https://www.youtube.com/watch?v=AsRrQLY5s_8

>>7478906
>I mean, they are probably freaking out over there, and started recovering from a backup, thought they could do it a quicker way and cancelled the original recovery, realized the second way doesn't work, and had to start over.
I work in IT (being intentionally vague here) and this kind of shit is super common.

>>7482815
>and this kind of shit is super common.
... especially when your dumbfuck business degree boss is bearing down on you and makes the executive decision to go with the "quicker way" even though the original way is working but will take awhile and you tell hid dumb ass that switching will just take longer--especially if the second, "quicker" way fails

i also work in IT and i'm a senior network/database admin for a multibillion $, multinational company. that dumb fuck boss of mine cost the company over $30 million immediately + probably $50-100 million in customer loss and tried to blame me. i got him fired by telling the boss-boss that he wouldn't listen to me and fucked everything up

>>7482955
>>7482815

You don't have to work for a multibillion IT company to know that sometimes you fail at predicting or picking up the best recovery method.

PHB is another risk thats true, but I trust crypto exchanges to be more stacked on nerds than on PHBs right?

>>7483054
>company to know that sometimes you fail at predicting or picking up the best recovery method
i told him that what we were currently doing would work and that it was the best method.

again, i got his ass fired. i was ready to be fired, myself, when i was summoned to explain the situation to Biggu Boss so i literally opened with "he fucked everything up by not listening to me" and that's what saved my job. i've learned that higher-up businessmen are absolutely no bullshit kinda guys

>>7482955
>>7483054
Often times the people who make it into decision making positions are bean counters. Bean counters are impossible to reason with when talking about IT, because companies view IT as a net loss, a direct hit to their bottom line that contributes no revenue. It's very very difficult to get people to understand "what if", and convince them why redundancy is important for tomorrow, when everything is working fine today. Even down to the end user level, often times inexperienced people tend to have high risk tolerance when it comes to IT because they haven't experienced long term service interruption, or god forbid the dreaded data loss scenario. For some people (looking at binance) it's a lesson that can only be learned the hard way, and that's super unfortunate.

>>7483210
Unfortunately, no-bullshit types are not the only ones who make it up that high, and you have to gauge very carefully who you're dealing with, because the one's who want bullshit (egomaniacs) wont stand for straight shooting and that can get your ass canned if you're not careful. Fortunately, it's easy to figure out which is which, because no-bullshitters will cut to the chase very quickly, or just straight up tell you not to bullshit them when they detect it (often they have good sense for bullshit), and the egofags will usually fluff you up with their own bullshit to basically indicate their expectation of how they want to be talk to. Best course is to be a straight shooter and dial it back when necessary.

>>7483793
>the egofags will usually fluff you up with their own bullshit to basically indicate their expectation of how they want to be talk to
you've opened my eyes senpai. looking back on conversations i've had...

>>7478580
Software developer here with experience in CRM databases. Data recovery should be fairly quick if they have a system in place to deal with these kinds of scenarios. The fact that they're having trouble means that their backup system isn't automated and they have to take manual steps to restore and validate the data.

>>7484109
I'm actually not a very skilled IT technician. I'm a people person, I know people very well and communication is the most important skill you can have in IT next to troubleshooting fundamentals.

Ironically, in the Information Technology field, which is literally based around making communication easier and more efficient essentially, communication continues to be the biggest problem.

>>7483210
>>7483793

Ahh muh M$hotgritbeowulf faggots. I knew that baiting you here was worth it. Hope you are having fun.

Yeah I've been learning from you (for years) how fucked and assinine can be the IT industry, just that this is another culture and another niche, I was expecting better outcomes with issues like this.

So whats the plan for when Binance goes online again? In other thread they are talking about dumping everything and fishing fatfingers.

>>7484406
>So whats the plan for when Binance goes online again?
idk and idc. i'm buying more BTC when/if it hits $4k and that's my rule

>>7483210
>"he fucked everything up by not listening to me"

wew

>>7484460
like i said, i was already expecting to get canned so i just shit out the truth

>>7482339
you win the thread anon

>>7484482
did you really use those exact words?

>>7484406
I've no idea what you're on about you silly cunt.

I have no personal knowledge of Binance's situation except that they weren't prepared for whatever happened, which likely means that some bean counter decided it wasn't worth the extra dosh to have a hot failover ready to go.

>Why do we need two identical servers??

Although, I also heard they were running on AWS, which kindof throws that theory out, I guess?

>>7484563
yes. it was the first thing i said

>>7484482
>the truth
You have no idea how correct this method is. Even when it might seem like the truth is going to hurt you. Good people see when people are telling hard truths and appreciate it. Only liars don't appreciate the truth.

>>7484622
Hit and miss
>>7484581
Yeah it looks like it's AWS so thats why I've been wondering the whole thread

> Well yes, the scenarios these nerds paint are all fine and non bizarre if binance weren't using AWS

I just checked Tabtrader, even the basic API services are dead. Last time Binance crashed I could still see/trade my shitcoins in tabtrader.

I don't want to FUD cause I somewhat like Binance but...

> Turn it on when Americans are slept.

My 0.0005 satoshis say that they tried some node js / angular upgrade and as usual this shit doesn't work as expected after updating some modules.. so back to working v1.0

Meh, gotta give the guys some time not to rush too much and do things the right way. Cause its the only way.