Hey guys, I'm a fellow developer of distributed systems here.
First of all I think what you are doing is great.
My question is what's the point of clocks at all? The current time is a very subjective matter and I'm sure you know this, the only real time is at the point when the cluster receives the request to commit. Anything else should be considered hearsay.
Specifically the time source of any client is totally meaningless since as you say further in the discussion that client machine times can be off by huge margins.
If you accept that then one has to accept the fact that individual machines within the cluster itself are prone to drift too, although one can attempt to correct for that I appreciate.
Wouldn't you think though that what is more important is that the order is more based on the bucketed time of arrival (with respect to the cluster).
I don't see how given network delays anyone can be totally sure A is prior to B, atomic clocks or not.
What is important is first to commit.
[edit] Yes would love to talk privately about this topic @irfansharif
When a single system is receiving messages, you pick an observed order of events that meets some definition of fairness, and you stick with it all the way through a transaction. By pretending A happens before B (even if you're not entirely sure) you can return a self-consistent result. And once you have that you can simplify a lot of engineering and make a lot of optimizations, so that the requests aren't just reliable but also timely.
You throw three more observers in and how do you make sure that all of them observe the requests arriving in the same order? Not even the hardware can guarantee that packets arrive at 4 places in the same order, even if the hardware is arranged in a symmetrical fashion (which takes half the fun out of a clustered solution).
> Specifically the time source of any client is totally meaningless since as you say further in the discussion that client machine times can be off by huge margins.
distributed systems like cockroach shouldn't use the client's conception of current time for anything at all, except possibly to store it (_verbatim_, don't interpret it) and relay it back to the client or to other clients (and let the client interpret it however they want).
Hmm, I'm not sure I completely understand your question or your source of confusion here but unless I'm grossly misunderstanding what you're stating I think we might be conflating a couple of different subjects here. I'm happy to discuss this further over e-mail (up on my profile now) to clear up any doubts on the matter (to the best of my limited knowledge).
Actually, WalMNart cares and so does T-mobile. You probably care too if you stop and think for a bit...
The concern here isn't just order of transactions, but also synchronization. For instance, WalMart might charge you twice for a transaction if it appears to have happened at different times when it arrives in different data centers.
Also, the comment "The higher frequency the transactions the more you get into quantum physics." isn't relevant here. This is more in the realm of relativity than quantum physics. Even so, we aren't currently at a point where we need to worry about transactions happening at relativistic speeds.
Ah, I think I see where you are confused -- your arguments seem to make more sense when dealing with a single, local database. The idea here is that you want to achieve atomicity, but you need to do it across multiple distributed databases and you want to have a system who's components have exactly the same time in order to ensure consistence across each database.
Attempting to extend the landlord example... let's say that I'm your landlord and you have to pay me £1000 each month. You send the bank a message telling the to pay me the money. The bank may make several copies of that message and keep it around for their own reasons. Now, let's say that there are employees at that bank whose job it is to do go through all copies of all messages and make sure what they say is done. If they find a message from several months ago saying "transfer £1000 from you to me this month" and are somehow oblivious to which month it is, they may transfer an additional one thousand pounds even if it's already happened. It's not an exact analogy, but...
Instead of just downvoting, how about refuting my claim?
I'm seriously curious what is the disagreement. These guys already established atomic clocks are unnecessary. Very interested in which use cases require them.
Serializability is all about ensuring a single consistent ordering of events. Lots of algorithmic shortcuts you can take if all your nodes' clocks are precisely in sync.
I'm very familiar with the literature since I'm a distributed database developer.
If you investigate high frequency trading you will understand that the quantum phenomena that I'm talking about is not just me high on mushrooms but a real world thing.
The only "time" relevant is the time when the cluster agrees an atomic, isolated transaction is time to commit from its own perspective.
Am I wrong in remembering that the HN guidelines used to say that you should not downvote someone's comment simply because you disagreed with it?
I went looking, and I don't see that in the current guidelines. I could be wrong about it being there before, but I was almost certain that it was at one point.
Seems like it used to say that you should only downvote comments that you think don't contribute anything of value to the conversation.
Just curious, because it seems to me that for quite a while now there have been a lot of comments that appear to get downvoted just because people don't agree with what the person said (and often there are no responses to counter, the person just gets downvoted).
I don't have a full history of the guidelines, but the canonical link on this tends to be [0]. About nine years ago, PG thought downvote to disagree was perfectly reasonable. I don't think there's been any official change since then.
I think you're thinking of somewhere else. The up/down votes are a way of agreeing or disagreeing without cluttering up the comments with a bunch of "me toos" or "nuhuhhs"
I'm certain that I'm not thinking of somewhere else. I'm completely open to the possibility that I just remember it wrong, but I'm sure that it was HN that I was thinking of, and not another site.
There's never been such a policy on HN; you remembered it wrong, as have many before you. It's the same phenomenon as attributes pithy quotes to Einstein and makes Canadians think we have Miranda rights: people hang memories on the nearest pre-existing hook in the brain.
Nope. I've never spend much time on reddit, and I'm certain I've never seen that page before. Perhaps I am thinking of comments other people made on HN in the past (who thought that the policy was as I described).
Nope, I know I'm not thinking of Reddit (as I said in reply to another comment, I've spent almost no time on Reddit, and would not have seen their guidelines at all).
That said, I think many on HN do think that downvotes should be reserved for uncivil or unsubstantive comments as they don't contribute to the conversation. Some will still downvote for disagreement or for other reasons.
I think it's best not to let it bother you or worry about it because there's not much you can do about it, other than contribute as civilly, substantively, charitably, and in good faith.
I guess there are people out there who really believe the individualism in Atlas Shrugged is a model for society? My interpretation is that's darkly satirical humor.
Okay Ayn was a genius, but I think she had a sense of humor too. At least that is the way I would like to think of it.
Which reminds me, I just watched an excellent documentary on Karl Marx which was very timely. It's largely forgotten what he was really about I think.
Really hit home that his concept of the "Surplus Value" of a worker is just as important if not more so today as it was the day he critiqued capitalism in the 1800's.
Shame about how his work was twisted into authoritarian dictatorship by the likes of Stalin. You just can't win :/
Serious question: how do you think one _could_ implement Marx's solutions to the problems he claims he found, _without_ an authoritarian dictatorship? Not thinking so much small, voluntary communes but at the sort of scale of a modern city or State.
I'm pretty sure it was Marx himself who first used the phrase "dictatorship of the proletariat".
People talk about Marx as if he wrote some kind of blueprint for a Communist state. But he really didn't. He was basically writing a philosophy of history centering around a problem he observed - he wasn't necessarily writing about ideas for implementing particular solutions beyond vague notions that a Socialist revolution and subsequent Communist society is ultimately inevitable due to built-in flaws of any Capitalist system. The TLDR version of Marx is:
"All of human history is based around class struggles punctuated by revolutions, and in modern times this continues with those who own the means of production (Capitalists) having all the power over the common workers (employees). But don't worry because Capitalism is a self-destructive system, and soon there will be a socialist revolution, which will ultimately evolve into a Communist state, and then everything will be cool."
The only real implementation details he talks about have to do with some kind of transitional "pre-Communist" society, where he talks about nationalization of banks and railroads and other things.
I think Marx is better read as a philosophy of human history rather than as a blueprint for creating a Communist society.
The reality is Marx is right in the sense that the "ownership class" (those who own the means of production) continue to accumulate wealth and project disproportionate power because they own large amounts of corporate stock and by extension have control over the physical (or digital) means to keep making money. However, in the Western World, no revolution was forthcoming due to (I conjecture) rising standards of living and a consumer-oriented society, along with a blurring of the lines between the "ownership" class and everyone else (via things like stock options, entrepreneurship, Unions, employee rights, etc.), which certainly diluted Marx's eternal class-struggle narrative. Still, at the end of the day, we find ourselves in a Capitalist world where the majority of wealth is concentrated in < 0.0001% of the overall population, which seems to be an undesirable situation.
Has there been studies/hard numbers that would model a world where the top 1% had their wealth more distributed? I'm not talking about pure socialism (equal wealth), I'm suggesting a more bell-curve-like distribution of wealth, and only in western first-world countries. Would the quality of life be (much) higher?
It was much higher in the U.S. decades ago before the process started where the big companies cut employees and maximize profit for shareholders at the cost of everything else. The layoffs and offshoring alone reduced quality of life for many people. Then there's mega corporations like Walmart that can afford to pay people enough to pay their rent or buy food but intentionally don't. You'd see a bit more competition (and jobs) in mainframes, desktops or mobile if threats to incumbents didn't get hit with their patent or copyright suits. As in, legal monopolies designed to maximize profit of owner at everyone else's expense.
So on and so forth. The centralization of wealth and power into a rich few running a bunch of oligopolies provably harms the many in this country. Directly when there was immediate layoffs, low quality/security, bad services, etc. Then indirectly in long-term and network affects.
> Still, at the end of the day, we find ourselves in a Capitalist world where the majority of wealth is concentrated in < 0.0001% of the overall population, which seems to be an undesirable situation.
That doesn't necessarily follow, for reasons you allude to. Massive wealth inequality is a lot more tolerable, if the _poor_ people are (by historical standards) incredibly wealthy.
Size is definitely a problem. Bertrand Russell touched on some of the issues it creates (e.g. signal - votes from the people effected by an issue - gets swamped by noise - votes from people who don't really care).
Personally, I'm a big fan of a couple of approaches that I think would work well in concert:
* Federalism. Devolve as much power as practical down to the State (or, as I suspect will happen, the City-State, level). Taking the USA as an example (I'm not American myself), you'd pare back all of the unconstitutional three-letter-departments (FCC, FDA, EPA, ...) and leave those up to the States.
* Sortition. Don't select the Government by voting, select it by random poll of the citizenry, in much the same way as jury duty is arranged in many countries.
* Libertarianism. Legislate to protect rights and in the case of collective problems (e.g. atmospheric pollution, which may surprise people by being the go-to example Rand reached for as a case where environmental legislation may be justified by human costs), but nothing else.
Marx''s "solution" was to let Capitalism run its course and entirely crowd out labor with capital, to a point of near-infinite supply and intolerably low demand. Over 100 years later, this has yet to happen. And people generally sympathetic to masdive redistribution call the Laffer Curve "voodoo economics"... Hah!
>Not thinking so much small, voluntary communes but at the sort of scale of a modern city or State.
Funnily enough, I remember going to some talks about something called "Economy of Communion", to which an audience member asked a similar question.
The response: think globally, act locally.
I wish I could remember more about the explanation, and although I remember not being completely satisfied with the answer, I also remember thinking that it made sense and it's probably better than the alternatives.
Hmmm, well as I see it, Marx, like many luminary geniuses forgot about one thing.
Human Nature.
The ideas in and of themselves were profound but the prescription was unfortunately childishly naive.
That given sufficient resources and everyone pulling together we will all get along is great, but it just takes one self-interested outlier to spoil that soup, and there are plenty of those to go around.
Some people can just never have enough.
Personally I believe self interest is pre-programmed at the genetic level in all species. Dawkins wrote a book about that, and it manifests itself in everything we do individually or across species.
So back to your question, that's a good one.
The only way to make it work is probably some kind of authoritarianism, but perhaps authoritarianism comes in many flavors.
I'm highly suspicious that a single individual can do it. Even if they are a benevolent dictator they can be corrupted by the people who surround them. It really doesn't take much to Gaslight someone and it happens a lot more and in more subtle ways than many of us think.
So what is a more pure form of a benevolent dictator. A computer program with hard rules that we agree upon?
At least it would be consistent.
Then again, the program would be written initially by our own genome, and it better be pretty clever to quickly figure out how to ignore that aspect.
But just imagine it ,a logical machine that only does the right thing to maximize outcome... (Already sounds like the Genome)
Planet in jeopardy, insufficient resources, no way to create more resources? Simple answer, kill a percentage of the populace.
Looks pretty Stalinesque already. But I guess you already know that.
It's a problem for sure.
There are radical ideas to sidestep it, but those would take more than a spoonful of sugar I think to get the medicine to go down.
As far as dictatorship of the proletariat...
"In Marxist sociopolitical thought, the dictatorship of the proletariat refers to a state in which the proletariat, or the working class itself has control of political power."
The big mistake I see in HN is that the members believe themselves to be elites, which for the time being may well be true, and many here are making hay. Probably a wise move.
The truth is we are also soon to be no more than axle turners in wheel factories, and many of us already are.
I'm not a dystopian, I really am an optimist. Hence the desperation.
Isn't there some middle path? Maybe the reason more and more sound like they want to go in a socialist direction isn't that we are all a bunch of socialist, rather, the pendulum has swung so far in favor of inequity in the USA the reaction has the appearance of socialism.
Many of us are waiting for the day that the wealth is spread to the outer reaches of the populace, those who have not. I suspect, we'd see a huge explosion of wealth for both less and more wealthy people. Here I'm not just talking about dollars on the bank statement, but a happy, enjoyable society to spend you time in. You don't have to step over the homeless person, for example.
Stop concentrating wealth, what does that mean? You're just going to give people stuff? Yep, you get a phone, you get shelter and everyone gets a bed, meal and healthcare.
Well that's how I run my life and I know a lot of other people do that as well.
We used to talk about having "Fuck You" money back in the day.
What was the amount of cash required to get off the system.
Turns out it isn't that much, and it would be even less if the system was geared around that mentality.
I need groceries, some nice olive oil a nice clean bathroom, perhaps a bottle of wine and watch a couple of movies.
Just provide that to everyone and lay out the propaganda to set the bar that this is all you need.
If you want more than that, taxation should go through the roof on an exponential scale, and not to be fed into the military, but fed to others to get them to the same basic standard.
Anyone with billions in their bank account should be very embarrassed by that fact, and I hope they are.
Knowledge of NLP helps. Not practicing it consciously but understanding the theory behind it.
Is important to make a connection and pay attention to others. Too many people make it a one way conversation and walk away wondering why things didn't go better, meanwhile they were talking 80% of the time.
Any time you can make others feel included, and their opinion matters during a meeting the better things will go.
I studied a little bit of NLP, and definitely think it has the potential to be useful. But in reality, I rarely, if ever, use the stuff (at least not consciously).
Right, NLP as a "Science" is kind of crackpot, but there are some things in the texts that are truisms and worthy of note just because they are laid out concisely.
It can be very dangerous and malicious to use that basic knowledge of how to conduct a conversation to do what? .. "make people feel special and you care when you absolutely don't give a damn about them", and the conversation is purely a means to an end.
That said, it's nice to be nice, and much better to be inclusive, even if you needed to learn that from a pretty fringe bunch of theory.
[Edit] As a side note, I was first introduced to NLP when someone used it on me in person as a means to try to get me to work for an organized online credit card phishing operation and wanted me to move to Costa Rica.
It was a very strange experience and was effective up until the point I noticed what they were doing
Took me a few days figure out what had happened. I didn't know anything about NLP at the time and was kind of shocked that this was an actual thing.
I really don't want to get into it, but it involved a beautiful Russian agent driving a BMW X5, baking implements and cup cakes. That's how it started.
Not the first time I've been approached by the Russians and to be clear I have nothing against them either, but if and when they do it again, please don't steal my paper towels or abuse my washing machine.
Right so reading that you probably think it is so fantastical that it can't possibly be true, or that it is so bizarre what I wrote that it makes no sense and I'm nuts. But there are people out there who know what happened and the truth is even more strange than I can share here.
Well I would agree with you on the weirdness, it was fantastical and it was bizarre, but it isn't the first time I've been approached by them or probably the last. The first time was in the early 90's just after releasing my DES hack thing. It just got weirder from there.
Honestly quite like the attempts in some ways, it's flattering, and she really was talented as were her friends. Cute too.
So to them, please go ahead and try again but next time, leave the Costa Rica thing out, or at least make it somewhere a little more temperate.
I really have no problem working for the intelligence agencies as long as it is for a good reason.
It's nice to see you've reached out to Alex publicly.
Any time someone or an organization brings in someone else's code and makes use of it, no matter what the license we really need to recognize that, it's just the morally responsible thing to do.
As far as ourselves, we're definitely using uWS and that is not going to go without recognition.
Nobody should need to "open a ticket", send an email or post on HN to get that sorted.
npm install and git clone making it way too easy for people not to give a crap about the works of others.
Good to see you guys are doing the right thing, uWS is a serious piece of tech and well worth recognition.
The fact that you're offended does not make a comment rude. It makes a valid, interesting point, that disdain for rules can be common to genius and criminality.
In fact, I think your comment is exceptionally rude and disrespectful. If you're not prepared for viewpoints to be challenged, stay out of discussions.
I'm not offended, I'm stating a fact that we were talking about what makes a genius and you decided to make it about rape and there was an implicit derision aimed at the person you replied to in addition.
>I'm stating a fact that we were talking about what makes a genius and you decided to make it about rape
Conversations move and branch out from their starting subject. That's in the nature of conversation.
But even keeping it on the same subject, sometimes people bring an example or a metaphor from a different area to clarify something about the subject. And sometimes they try to use something obvious or universally agreed as a good or bad counterexample to drive a point home. Which is exactly what happened here.
If we talk about computing and I bring an example from baking, I don't "make it about baking" anymore than the parent "made it about rape". If anything, the parent just made a single comment that just happened to use rape, he could have used any number of similar examples, e.g. arsonists or flat earth believers or Drake fans -- you fixated on his use of the word rape.
>there was an implicit derision aimed at the person you replied to in addition.
Only an imagined one. The only thing the parent did was to aim at a gaping hole in grandparents definition. No derision towards the person itself.
I apologize that you (or anyone else) are offended -- I didn't intend my comment to have anything to do with rape, or to imply anything.
We were discussing the definition of a "genius" and I was trying to point out a problem with the definition proposed. I feel that's perfectly in line with the "narrative".
Any number of unacceptable labels/behaviors could have been used as an example here.
I actually knew what you meant and probably the other person did as well.
It did come off a bit harsh, that was my only point and I don't think think it was offensive but it seemed like it was conflating two very different things that should have no "nearest neighbor search" with each other.
Yes, definitely agree on the last point.
What makes a genius -> My personal experience -> Rape.
I'm not sure how much more diplomatically I could have phrased my previous post. If mentioning a language as one possible modern alternative ("languages like Rust") is enough to set you off, perhaps you need to reconsider who's falling short of adulthood here.
First of all I think what you are doing is great.
My question is what's the point of clocks at all? The current time is a very subjective matter and I'm sure you know this, the only real time is at the point when the cluster receives the request to commit. Anything else should be considered hearsay.
Specifically the time source of any client is totally meaningless since as you say further in the discussion that client machine times can be off by huge margins.
If you accept that then one has to accept the fact that individual machines within the cluster itself are prone to drift too, although one can attempt to correct for that I appreciate.
Wouldn't you think though that what is more important is that the order is more based on the bucketed time of arrival (with respect to the cluster).
I don't see how given network delays anyone can be totally sure A is prior to B, atomic clocks or not.
What is important is first to commit.
[edit] Yes would love to talk privately about this topic @irfansharif