Gee, I wonder why so many people are so skeptical about 'official' studies and reports these days that often seem to push a particular political agenda.
The author is right. Too many in the established media are not at all concerned if something is accurate or not. All is has to do is agree with their preferred narrative.
Sometimes it feels like there is an awful lot of software out there that shipped without much review. This was happening long before AI arrived on the scene.
Hard to tell if anyone was 'comfortable' with that.
The open source world is littered with all kinds of specialized licenses that came about because a developer once said "Here is my source code. Do anything you like with it! ... wait! Except for that!".
I don't know if there is, but I wouldn't be surprised if there is a license available that restricts you from porting the code to MS Windows or any other proprietary OS.
This article points out some of the pain associated with index maintenance. It should also point out that ALL indexes on a table suffer from the same issue. If your 20 column table has 7 indexes, then the suggestions should be applied 7x.
It is conventional wisdom that indexes are absolutely essential for any relational table of at least reasonable size (e.g. thousands of rows) and is accessed more often than daily. Indexes can be a pain to create and maintain; but can greatly speed up queries and primary key validations. The pain mostly comes from having to figure out what indexes to create and how often to maintain them, rather than doing the actual thing.
Indexes also have a performance penalty for any table updates. Creating new rows, updating existing rows, or deleting rows all require updates to each index.
But are indexes really required? I am creating a new kind of general purpose data management system (a kind of object store) called Didgets. The tagging mechanism that I invented to allow tags to be attached to each data object, are key-value stores that essentially form a set of columnar stores.
I found that these columnar stores could also be used to create regular relational database tables. The data is structured such that indexes are not needed. All the tests that I have run (up to a thousand columns with over 100 million rows), show that query speeds are equal to, or better than other database systems that are well indexed.
The system is still under development, so it is still missing some key features that would make it a drop-in replacement for other databases; but it proves that it is possible to structure relational data such that query speeds can be optimal without needing separate indexing structures that have to be maintained.
I’m not clear on how you’re deviating from a normal columnar/OLAP database?
> I found that these columnar stores could also be used to create regular relational database tables.
Doesn’t every columnar store do this? Redshift, IQ, Snowflake, ClickHouse, DuckDB etc
> but it proves that it is possible to structure relational data such that query speeds can be optimal without needing separate indexing structures that have to be maintained.
Doesn’t every columnar database already prove this?
I am not an expert on all the other columnar stores out there; but it is my understanding that they are used almost exclusively for OLAP workloads. By 'regular database tables', I meant those that handle transaction processing (inserts, updates, deletes) along with queries.
My system does analytics well, but it is also very fast with changing data.
I also think that some of those systems (e.g. Duckdb) also use indexes.
They’re used by OLAP workloads because columnar properties fits better — namely, storing data column-wise obviously makes row-wise operations more expensive, and column-wise operations cheaper; this usually corresponds to point look-ups vs aggregations. Which cascades into things like constraint-maintenance being more expensive, row-level triggers becoming a psychotic pattern, etc. Column-wise (de-)compression also doubles-down on this.
They still do all the regular CRUD operations and maintain transactional semantics; they just naturally prefer bulk operations.
Redshift is the most pure take on this I’ve seen; to the point that they simply don’t support most constraints, triggers and data is allocated in 2MB immutable chunks
such that non-bulk-operations undergo ridiculous amounts of write amplification and slow to a crawl. Afaik other OLAP databases are not this extreme, and support reasonable throughput on point-operations (and triggers, constraints, etc) — in the sense that it’s definitely slower, but not comically slower. (Aside: Aurora is also a pure take on transactional workloads, such that bulk aggregations are comically slow)
> I also think that some of those systems (e.g. Duckdb) also use indexes.
I’m pretty sure they all use indexes, in the same fashion I expect you to (I’m guessing your system doesn’t do table-scans for every single query). Columnar databases just get indexes like zone-maps for “free”, in the sense that it can simply be applied on top of the actual dataset without having to maintain a separate copy of the data ALA row-wise databases do. So it’s an implicit index automatically generated on every column — not user-maintained or specified. I expect your system does exactly the same (because it would be unreasonable not to)
> My system does analytics well, but it is also very fast with changing data.
Talk more, please & thank you. I expect everything above to be inherent properties/outcomes of the data layout so I’m quite curious what you’ve done
My project Didgets (short for Data Widgets), started out as a file system replacement. I wanted to create an object store that would store traditional file data, but also make file searches much faster and more powerful than other file systems allow; especially on systems with hundreds of millions of files on them. To enhance this, I wanted to be able to attach contextual tags to each Didget that would make searches much more meaningful without needing to analyze file content during the search.
To facilitate the file operations, I needed data structures to support them. I decided that these data structures (used/free bitmaps, file records, tags, etc.) should be stored and managed within other Didgets that had special handling. Each tag was basically a key-value pair that mapped the Didget ID (key) to a string, number, or other data type (value).
Rather than rely on some external process like Redis to handle tags, I decided to build my own. Each defined tag has a data type and all values for that tag are stored together (like column values in a columnar store). I split the tag handling into two distinct pieces. All the values are deduplicated and reference counted and stored within a 'Values Didget'. The keys (along with pointers to the values) are stored within a "Links Didget'.
This makes analytic functions fast (each unique value is stored only once) and allows for various mapping strategies (one-to-one, one-to-many, many-to-one, or many-to-many). The values and the links are stored within individual blocks that are arranged using hashes and other meta-data constraints. For any given query, usually only a small number of blocks need to be inspected.
I expected analytic operations to be very fast, like with other OLAP systems; but I was pleasantly surprised at how fast I could make traditional OLTP operations run on it.
I have some short demo videos that show not only what it can do, but also benchmark many operations against other databases. Links to the videos are in my user profile.
Interesting ideas. Im very interested in database ideas that bring new capabilities or better ways to acconplish old ones.
W.r.t. query speeds on your columnar storage engine, you will obviously have much better writes that row oriented storage engines. This limits your write capabilities though. Any effort you put into restoring write speeds necessitates an extra step to the maintain the columnar stores - which puts you back into the group of databases naintaining indices that you criticize above.
I think modern databases are bringing new ideas on how to accelerate both write and query speeds simultaneously with tradeoffs around CAP.
Although I have done many more benchmark testing against other databases for query speeds; I haven't noticed any significant speed degradation on writes.
Could you clarify what you mean by 'this limits your write capabilities'?
> W.r.t. query speeds on your columnar storage engine, you will obviously have much better writes that row oriented storage engines.
This should have said reads, not writes. Columnar storage takes significantly more effort to handle writes because it must do many more IOs across the different columns, potentially more de/compression cycles, etc.
This guy lost me when he started talking about diversity issues instead of tech. I couldn't care less about the race, gender, or sexual orientation of the person(s) who created the hardware or software that I use. Does it work? Is it easy to understand and use? These are the things I am interested in.
I am reminded of an early cartoon of a dog sitting at a computer saying 'On the Internet, no one knows you are a dog!'
For me, it's an especially bad argument because the sloppy nature of HTML parsing is NOT a virtue... it's a source of bugs, vulnerabilities, and incompatibilities that provides (yet another) technical moat for existing web browsers. It's a huge tragedy that HTML5 beat XHTML.
Maybe, but after several decades of engineering, I became much more of an advocate for actual observed user behavior, which is chaotic, and as such nowadays preach “embrace chaos”.
That’s what HTML and the browsers did. They accepted humans are terribly bad at following instructions when you want to cater to a broad audience, and as such embraced error correction. The end result is that the early days were awesome because everyone knew how to build websites.
Perhaps in the current day and age, where people hardly write hand-written HTML anymore, this can be reconsidered. But a new, more restrictive format would have to show real benefits, because it’s precisely as I say: people don’t really write raw HTML anymore so it’s kind of a moot point.
You do care, otherwise you would not even have thought about it twice, let alone comment.
The most striking argument against this line of reasoning is that there is no possibility for you not to act politically. By ignoring the issue of systemic discrimination (“don’t care don’t personally discriminate”) you actively contribute to and participate in the systemic discrimination.
It’s a choice, and I find it important to leave you the choice, as I do not believe forcing anything on you will actually make you see, but it is true nonetheless that you cannot escape your responsibility: The ability to respond. “You can not not communicate.” Not responding and deciding to remain ignorant about it is also a response, and a highly privileged one.
It would be easy to at least passively support anyone’s attempt at trying to reduce systemic discrimination, but speaking out against it turns it even more into a political act supporting discrimination than doing nothing and by that delegating it to others.
I have a hard time following this line of reasoning. I don't think GP's point is to "ignore systemic discrimination". Rather, it is to ignore those things on which people are discriminated. How can you discriminate against, say, race, if you don't pay attention to the person's race?
Specifically, for tech, if you happen upon a code written by someone called "didgetmaster" on github, and the author makes no comment about who they are as a person, how does this contribute to discrimination? Isn't this the whole point of anonymous resumes and such to fight discrimination against minorities for employment?
The issue arises when people disagree on what reality is.
When most Americans are running on pure ideology and a (quite unearned imho) sense of moral certainty and superiority, they assume their worldview is the objectively correct one, and everyone who disagrees with them is "a bigot."
Both sides of our divide have some psychotic people who do things like murdering people. But absent actually harmful acts like that, disagreeing with you doesn't make someone intolerant. It could be that your framing is wrong, or that there isn't one black-and-white objective universal right way of framing an issue.
> The most striking argument against this line of reasoning is that there is no possibility for you not to act politically. By ignoring the issue of systemic discrimination (“don’t care don’t personally discriminate”)
This is not a very good argument, it would make the words "political" and "systemic" simply meaningless. First of all, it's implicitly assuming that the only way of effectively addressing the issue of systemic discrimination against X is to actively care about whether X is involved in the stuff we personally use. But "personal" and "systemic" issues and approaches are mutually contrary; that's what "systemic" means. Also if every social action is "political" by definition, what's even the point of resorting to the word "political" to begin with? It's not saying anything of relevance.
To illustrate, say you live inside a fenced-off city. You say you don’t discriminate, anyone is welcome to come and talk to and trade with you. Somebody points out that there are people outside the city, behind the fence, that aren’t able to come talk to you. You are free to act or not act on that, but speaking out against the one that merely points it out and tries to change it means you take an active position to support the current discriminatory situation, rather than a passive, opportunistic one that supports whatever the political situation happens to be.
All positions are valid positions to take. They do however reflect an active choice and an active act. All of them are political. All of them come from a position of privilege, being inside the city, not outside.
No-one is "speaking out against" the one who is pointing out any wider discrimination, beyond whatever aribtrary circle you choose to draw.
What is being "spoken out against" is the idea that taking the moral (or political) action within whatever circle you feel able or willing to support is insufficient, or even discriminatory in itself. After all, this is exactly how this conversation started. Good for you if you want to change the world - let's not forget 3rd party discrimination against other 3rd parties! For many of us, it's one of numerous pressing problems to be addressed. If you wish to bring privilege into it, having the freedom to make fighting any and all discrimination a primary concern is a sign of privilege that few have.
I am raised in hindu religion and we have a saying here basically meaning, treat others the way you wish to be treated yourself and I am sure that literally every main religion and philosophy can kind of share the sentiment and create a tolerant society overall.
Shame that those same religions and people in power forget this core part I suppose. I think that the forgetfulness might be on purpose but I basically hope we can treat literally everyone others the way we wish to be treated ourselves, with dignity and respect.
Of course, some people who are clearly bad shouldn't be treated this way but I hope you can get what I mean by the general sentiment of this idea I guess.
Wow, I had just observed that this seemed very common in religions I didn't know I was this accurate as that was called the golden rule!
Now hearing it, I remember hearing something similar but thanks for refreshing and thanks for letting me know
Now that we are talking about golden rule, I often wonder, if this is the case, then why not skip all the other aspects of religion which were meant in tribal times or have issues which still persist to put more emphasis on this "golden rule"
So your belief is that anyone who would claim to treat everyone the same falls into that camp?
That's quite the characterisation to make, and a bit difficult in the context of discrimination which is based on lumping people together based on one trait and making assumptions about everyone in that group.
> People who claimed they treat everyone the same turned out embracing openly fascist, misogynistic and racist movement last 3 years.
No, we just left the Democratic party once you guys stopped being serious. Judging people based on the color of their skin instead of by the content of their character is just as toxic and evil no matter what your claimed motivation is.
Most people don't really endorse Trump or endorse everything the radicals on the right are known for, but I can see that it appears that way to the people pushing DEI and race-and-gender-based everything -- because a clear plurality have indeed rejected the Democratic party, resulting in them losing even when running against a corrupt buffoon.
It is a collective issue, and as such also the responsibility of each individual indeed. And silence can be one of the psychologically most damaging forms of violence, leading even to physiological damage when persistent.
I didn't get a far as that because I just read through the massive list of enormous security problems caused by loose HTML parsing and that's the exact opposite of cultural accessibility. It requires arcane knowledge and rare skill then to be competent and safe. Tearing down those artificial barriers is how we let more people in!!
One of the interesting things about the internet is that almost anyone who runs a web browser on their phone is using a processor designed by Sophie Wilson, a trans woman, to run a language designed by Brendan Eich, a guy who donated money to oppose gay marriage and then lost the spot as head honcho of Mozilla because of it. I often think about this and wonder if the two of them ever met, or what would have happened if they did.
The internet doesn't run on "tolerance"; it runs on actual, real neutrality, which enforces tolerance by making anything else impossible. You cannot possibly stop gay people accessing your website any more than you can prevent homophobes or drug dealers or AI startups or North Koreans or fascists. You can ban behaviour but it's impossible to ban classes of people without real-life action.
I'll gladly take the opportunity to avoid directly supporting a racist/bigot, or anyone who seeks to silence/oppress/murder entire groups of people for simply existing. I don't care if they're selling me an application, or a laptop, or a car, or a cheeseburger. The internet, and the world, are a lot more interesting and exciting when you get a lot of different people from different backgrounds participating. Genius and innovation can come from anywhere. Morally and practically I think we're better off being inclusive.
I may not know who on the internet is a dog, but I'm glad those dogs are out there and if somebody is a proud supporter of puppy genocide I'd rather not encourage/enable their misguided crusade so that does factor into my choices.
It isn't something that can always be avoided, or even something that needs to be avoided entirely 100% of the time (I still read Lovecraft for example) but I do think it's worth some consideration.
But where do you draw the line? Not buying a game that was created by the Nazi party to promote antisemitism is one thing. Shunning a text editor because one of its creators once said something you disagreed with politically, is another.
I tend to draw the line along the impact of my choices. If I download an open source text editor made by a Nazi for my personal laptop it's probably fine. The software is free so I'm not handing money to a Nazi who could use it to harm others. It's on my personal device so there's very little risk that I'm signaling support for the Nazi or advertising for his text editor either. Still, text editors are everywhere and if I can find one that works just as well for my needs and doesn't depend on a racist for updates that's probably a good thing.
I also don't the sweat the small stuff. Hideki Kamiya has been accused of being a xenophobic ass and maybe he is, but people are entitled to their bad takes and internet drama isn't going to stop me from enjoying games he's worked on.
> The internet, and the world, are a lot more interesting and exciting when you get a lot of different people from different backgrounds participating. Genius and innovation can come from anywhere. Morally and practically I think we're better off being inclusive
Agreed but honestly sometimes I wonder about the funding sides of things, I love tinkering with software and making (right now LLM based but I know it can be a crutch, I use it to just "test" ideas tbh) but basically my question is, I wish to do more stuff on the web but as someone from a third world country, even 30k $/year are enough to me right now.
I am extremely frugal, most of it might just go into my savings/ETF world funds but basically I worry that if I cannot earn that amount, or even if I do, I am subjected to some very high risks which scare me as I have seen its costs first hands
I want to build software which some people might even work on full time but I just don't know how to make money out of it. If I can't make money out of it, I just open source some half assed implementation of it and literally noone knows about it.
I don't know too much about marketing and monetization perhaps. I don't know how to monetize, I hate ads and don't wish to push them, most of my servers/services I create can be abused so there is an issue the server provider might block me and I don't know if people are willing to pay for the software.
I am in a unique position but honestly what most people do is that they create growth right now, then sell it or smth or take VC funding etc which might include enshittening in the future for profit.
Like I just want to build, and a sustainable/profitable business with a profitable business model but the web just lacks this ability and I think I try to see it from my lens where I am frugal so I expect everyone is so its :/
The internet is interesting if people participate from different nations but that means that I just feel like what I might do will always be risky and I always need a job to fall back upon I guess.
Maybe an compromise I think I will do is work in very adjacent industries and contact them directly and work for 30-40k$ so I can get paid "enough" for my country and basically be good to go I guess.
The thing I get worried about is that if any such business goes bad, I would be on the chopping block and that its inherently more riskier than any business here who will try to extract 10x more value out of me and be the middleman...
I wish to be a software engineer here out of my own passion but since literally everyone wants to be here, my talent literally means jackshit in colleges or their applications so Genius and innovation can come from anywhere but opportunities can have an inequality which I hope I can fix for myself and hopefully help others in solving too one day I guess Idk.
Except there are people right on this forum who will happily talk about how dogs deserve fewer/no rights. Do you think that this makes the "dogs" feel welcome and do you think that these words have no effect?
Like, it's cool and great that you personally are in a position where most of the ideologies of hate aren't affecting you, right now, personally, but is it too much to ask that you spare a thought for the people it does affect?
>I couldn't care less about the race, gender, or sexual orientation of the person
But that makes you a bigot and you know what happens to bigots so put on the shirt, label in your bio and change your speech or HR will have a field day with you. Welcome to the "tolerant" side.
Do you care, then, that people are driving away contributors specifically because of their race/gender/orientation? Because without them, we wouldn't have [the stuff in the article].
He talks about "throw[ing of] slurs" and "denying rights to others", "wishing violence on people because of their heritage", and that type of thing. Does not besetting people with slurs count as "diversity" now?
Do you think these people will be able to do good technical work if they're constantly beset by this kind of thing? Or do you think they will retreat to somewhere where they don't have to do with that?
And secretively hiding who you are is not a solution. No photographs. No video meets. No YouTube videos. Can never discus anything personal. No conferences.
> Does not besetting people with slurs count as "diversity" now?
It depends whom you ask! There are certainly some people both off- and on-line who seem to think that uniformly slurring all white, male, cis-sexual, whatever folks as "oppressors" - and even occasionally suggesting, apparently as a serious political proposal, that "Whiteness" (whatever that means) is something that should be forcibly done away with - is an effective way of advocating for diversity.
Hmm yeah it's quite badly written. Either way, let's just say it's an example of a place where one can look stuff like this up.
"Ending whiteness" does not mean "killing all white people" or "eradicating all European cultures" or whatever the parent comment was implying. (Or am I reading too much into it?) Though perhaps you can find some moron calling for that on Twitter.
Are other phrases of the form "ending ____ness", where ____ is a racial qualifier, also acceptable, then? Someone using that term could similarly argue that they're not advocating for the eradication of ____ people - just the identity and behavior associated with ____ people.
Are you arguing that calling it "whiteness" is a bad branding move by the sociologists? In that case, I fully agree. There's other similar bad branding moves like "toxic masculinity" or "racism = power + prejudice". I wish they would change those words.
Or do you mean that sociologists who speak about "abolishing whiteness" secretly mean "abolishing the behavior and identity of white people"? No I wouldn't agree with that. I think they mean what they say, and when they tell you their definition of "whiteness", that's what they're referring to. Not some other thing.
Going from "hey, maybe we shouldn't kick trans people in the teeth?" to some rant about "help help I'm oppressed because I'm white!" is quite the stretch. Why don't you just come out and say that We must secure the existence of our people and a future for white children? God this site truly is infested with white nationalists up to its tits these days.
It seems to me that there are many people in almost every field who are more interested in what you can do for them, than in working with you collaboratively to build something meaningful.
I don't know that there are more or less of them in tech, than in other fields.
I upgraded my pc about this time last year (new CPU, RAM, and motherboard).
I was originally going to just get 64GB of DDR5-6000, with the option of adding another 64GB later, thinking the price might drop even further. At the last minute, I decided to get the whole 128GB instead. Glad I did.
How many great opportunities are passed up by people waiting for the 'perfect' one to come along?
Job offers passed on because it wasn't the perfect fit. Marriage chances missed because the perfect soul mate must still be around the corner. Good schools rejected because it wasn't their dream school.
Luck often accompanies those willing to accept less than perfect in many cases.
Both Voyagers left the ecliptic plane with their final gravitational slingshots (Voyager 1 went north, Voyager 2 went south so only the Canberra radio dishes can communicate with it) so even when Earth is further from them than the sun there's 35 degrees of separation.
The author is right. Too many in the established media are not at all concerned if something is accurate or not. All is has to do is agree with their preferred narrative.
reply