Cython, pypy, micropython, nuitka, shedskin, ironpython, graalpython, jython, mypyc, pyjs, skuptjs, brython, activepython, stackless, transcrypt, cinder and many more I don't remember.
They're all practically useless or delegated to specific tasks. At this point you'd need to present incredible evidence that an alternative compiler can be useful. Personally I find it comical how many developers are still eluded by a promise of performant python. I hope you achieve your goals, good luck.
Solid writeup. From someone who does/did a lot of this professionally:
1. Android typically is easier for this kind of work (you don't even need a rooted/jailbroken device, and it's all Java/smali),
2. That said, instead of installing an entire framework like Xposed that hooks the process to bypass certificate pinning, you can usually just decompile the APK and nop out all the function calls in the smali related to checking if the certificate is correct, then recompile/resign it for your device (again, easier on Android than iOS),
3. Request signing is increasingly implemented on APIs with any sort of business value, but you can almost always bypass it within an hour by searching through the application for functions related to things like "HMAC", figuring out exactly which request inputs are put into the algorithm in which order, and seeing where/how the secret key is stored (or loaded, as it were),
4. There is no true way to protect an API on a mobile app. You can only make it more or less difficult to secure. The best you can do is a frequently rotated secret key stored in shared libraries with weird parameters attached to the signing algorithm. To make up for this savvy companies typically reduce the cover time required (i.e. change the secret key very frequently by updating the app weekly or biweekly) or by using using a secret key with several parts generated from components in .so files, which are significantly more tedious to reverse.
Yes.
You can reverse engineer apk-s to see what's going on. (apktool)
Or you can inspect the network traffic that goes through your wifi. (mitmproxy + Xposed/JustTrustMe)
You're right - their headline is written for attention. It's an exploit of a feature.
What I'm interested to know is whether there is any code already out there in the wild with this exploit in it? An intelligence service could have exploited this years ago without anyone noticing until now.
Unicode is a pathway to all manner of hijinks, including as you say, homoglyph attacks. For instance, on some TLDs I can easily create two different domain names that render identically in the browser.
There should be a better way to do that. Ideally I would want a tool shows Request Method, Headers, Query String, POST Payload of requests as they come in (and let's me filter on those). It should support HTTP2 and know how to stitch together the payload from multiple packets.
HTTPS is a difficult topic. I think it's fair for a tool like this, not to mess with encrpytion. Usually you don't want to had a precious private key file to a command line tool for debugging. A better way seems to be to terminate SSL on a separate host, and analyze the un-encrypted traffic.
I aggressively refuse to call myself an engineer. Programmer is fine. Developer is better, because programming is only a piece of it. Y’all can call yourselves whatever you like, but inside I’m thinking you retconned an industry insider euphemism into a non-existent sub-type of engineering, by protesting all the ways you apply rigor to what we do.
IMO, IPv6 sucks for home users and labs. Prefix delegation means your routable IP scheme could change underneath you, and firewalling connections using PD addresses is really hard (unpredictable prefixes, privacy extensions change suffixes constantly), though some routers make it easier, like Google Wifi. I think most will end up NATing a single external IPv6 with a private IPv6 network behind it.
I wish IANA and friends would let home users and other smaller entities request global /56 address blocks and force ISPs to peer/announce (nothing crazy, just a default route).
EDIT: end users should be able to request static blocks from their ISPs for free, not IANA.
I'm confused by this, because the websites with the most value to me are those with the fewest ads. Like this one. Or the company website for a tool I bought, where I can now go download the manual as a PDF for free. Or some personal blogs I read. Even Gmail, which used to be the classic "great service in exchange for ads", I only use through IMAP clients and honestly don't even know if they show ads in the web view any more.
If you mean journalism, that's a business that has been going out of business even with ads. Advertising is just a historical implementation detail for them, like "printed on newsprint", or "tossed on my porch every morning". It does not look like a serious contender for how to make journalism profitable in the long term, any more than "teenager bicycle delivery system" will continue to be the ideal distribution mechanism for weather reports and sports scores.
Not a tautology. Just insufficiently creative thinking.
Don't most of us here remember the olden days of the web? When internet advertising was a waste of time and most websites were funded from the pockets of the creators. I mean, even today that's partly common - how many bloggers and podcasters are asking for topup funding.
It's silly and wrong to think the internet would die without advertisers. Big content producers could make fairly priced single article access (don't you remember micropayments? instead we got subscriptions), and generally return commerce to the olden days when people anonymously purchased what they wanted from a shop, instead of signining away access to their bank account and email address (password collection points abound in the modern internet, deliberately or not, and everyone who encourages us to create a password is damaging the personal security).
Amateurs could use such platforms as well as advertising driven tools that are merely less effective at stalking you.
Really it's only the stalkers themselves who would be hurt by such a change.
Nothing really. I retained the right to sue for my own personal judgement, which I have not done.
The main thing I got was the satisfaction of writing a letter to the judge pointing out that the attorneys who were representing my class ignored my attempts to communicate with them. I dug up the letter:
To the Court for the High-Tech Employee Antitrust Lawsuit,
I am writing to explain my decision to opt-out of the proposed settlement for the lawsuit against my former employer, Google, and their collusion with other employers. I was a software engineer at Google 2001–2006.
Google, Apple, Adobe and Intel are some of the leading employers in my industry. The harm they did to software engineers, particularly junior ones, is enormous. It’s not simply the lost wages in the settlement period; the market distortion set back the careers of my friends. Artificially lower wages compound for years as new jobs’ salaries often depend on previous ones. Brilliant young engineers lost opportunities to advance their careers because they were unaware of opportunities at other companies. The harm done was significant and I do not feel the settlement is sufficient.
I’m also opting out because I am unimpressed with my nominated attorneys. The fact that Judge Koh ruled the original settlement was “below the range of reasonableness” suggests my attorneys are poor negotiators and have not represented the plaintiffs effectively. I do not see wish to participate in a settlement where they receive $80M+ dollars. Also personally, I’m irritated that they cannot even do class members the courtesy of answering email sent to the published contact address of **@****.com. My emails of April 29 and May 10 both went unacknowledged.
Thank you for reading my letter. I do not wish to speak at the Fairness Hearing.
> How are you able to keep the site running after HN kiss of death?
I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware.
My day job is also developing this sort of highly performance java applications, I guess it helps.
> What is your stack, elastic search or something simper?
It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries.
> How did you crawl so many websites for a project this size?
It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains).
> Did you use any APIs like duck duck go or data from other search engines?
Nope, it's all me.
> Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?
I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains.
It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.
> What crawler are you using and what kind of crawling speeds are you achieving?
Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper.
>How do you rank the results (is it based on content only) or you have external factors too?
I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query.
> What is your personal preferred search option of the 7 and why?
I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement.
> Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible.
And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing.
I'm self-hosting, and the server is a Ryzen 7 3900x with 128 Gb of non-ECC RAM. It sits in my living room next to a cheap UPS. I did snag one of the last remaining Optane 900Ps off Amazon, and it powers the index and the database--and I really do think this is among the best hardware choices for this use case. But beyond that it's really nothing special, hardware-wise. Like it's less than a month's salary.
It runs Debian, and all the services run bare metal with zero containerization.
Modern consumer hardware can be absurdly powerful if you let it.
Like I have no doubt a thousand engineers could spend a hundred times as much time building a search engine that did pretty much the same thing mine does, it would require a full data center to stay running and be much slower. But that's just a cost of large scale software development I don't have to pay as a solo developer with no deadline, no planning and a shoestring budget.
The root issue (pun not intended) is that DNS is not fully encrypted, not even point-to-point, DNSSEC only provides an integrity check, no encryption; DoT/DoH is only used between a resolver and a client as a last-mile solution. No robust security can be derived from the absence of full encryption - there is none, ultimately the traffic is sent in clear over the Internet backbone. You must make your own choice of trust for sending traffic in cleartext - your local network at home/school/hotel, your rented VPS provider, your ISP, or Google/CloudFlare.
Given that CloudFlare already has 13%+ market share, as an end-user, if my primary threat is eavesdropping in the middle, not eavesdropping by CloudFlare (or by NSA's subpoena or NSL), it's actually reasonable to trust the tech giants. Using CloudFlare's DoH server helps me to end-to-end encrypt my query of 13% of the domain names! Also, CloudFlare has business plan to adopt private encrypted DNS links to other authoritative servers, and they already have a private channel with Facebook, it's even better! Don't start sending hate replies, I know the bigger picture is that it allows CloudFlare to effectively monopolize DNS is extremely harmful in the long run, and I'm worried. But I simply don't see a perfect alternative solution that allows one to self-host a resolver at home with end-to-end encryption to communicate with authoritative servers.
You see, the end results that everyone should work towards is introducing encryption to the missing resolver-authoritative server link.
DNSCurve was developed in 2005 as a practical and high-performance protocol for deployment on authoritative servers, and fill the missing link, the protocol is great, DNSCrypt was its direct descendant for last-mile client-resolver encryption.
It was ultimately not adopted, partially because of many hate DJB's personal style of taking an aggressive position over technical issue, especially his attack on DNSSEC. But also, the final reason was that the ICANN has stated that, in the case of the DNS Root zone servers, DNSCurve will never ever be implemented. First because its threat model doesn't include the authoritative servers themselves, and it gives them the private key. This is fine for private use, but the DNS root servers are controlled by many political entities, and ICANN doesn't believe trusting them is acceptable. DNSSEC was designed in a way to avoid giving the ultimate key to root servers, but not DNSCurve. Also, DJB hated DNSSEC so much that DNSSEC cannot be used to sign the pubkey for DNSCurve resolver - oops!
In the future, we should work toward a solution for resolver-authoritative server encryption. In the article, the author mentioned DNS-over-TLS is a potential solution, if so, we should push to deploy DNS-over-TLS on major authoritative servers, or perhaps one day, on the root servers if a solution has been worked out.
Only then, it could allow us to say farewell to CloudFlare's DNS and Google's DoH resolvers.
Disclaimer: I work at Google. In cloud, not on Android.
I am privacy conscious so I though I would give a try at Graphene OS, it was brutal. It was overall stable and the stock android R was refreshing. The app selection available through F-droid was very limiting and the quality of apps was a struggle compared to those in the app store.
I wish there was some incentive for the better apps to open source and publish on F-droid. I donate to many apps on Patreon, should I just message to the devs and ask them to go open source and publish on f-droid?
The simplest explanation is that the performance benefit of using Windows and implementing just about any Windows-specific design is outweighed by the cost of the Windows licensing fees, when compared to a measurably worse-performing Linux or FreeBSD solution that costs nothing. So very few bother to treat Windows versions of "backend" software as anything but an afterthought.
This is just an anecdote, but I had to write a SQL DDL and DML parser during my last semester. I wrote the DDL parser by hand, and it wasn't as bad as I expected, but it was time consuming. I managed to convince the professor to give us the option of using a parser generator for the next phase (DML) since the point of the class wasn't parsing context free grammars and more focused on executing the SQL.
I used Flex and Bison since the project was in C. Getting up and running and understanding how the tools have to be set up with different options was a bit tricky, but after that, my parser was up and running in about two hours, compared to probably four times that for the hand written DDL. Our DML subset was also much larger and more complex than our DDL, so I was very happy with the development speed increase.
I had this idea that using a parser generator was slow and wasteful since many modern tutorials online write them by hand and speak against parser generators (possibly because there isn't a catch all for all languages). Turns out dev speed is way more important to me up front, because in the case that I notice parsing speed actually being an issue I should be happy that my MVP has gotten enough use.
It's also nice because a lexer and parser can be pretty easily black-boxed and swapped out for your hand written, just keep the AST and API the same and you should be good.
All that said, that's personal preferences and writing the parser by hand is definitely good experience and more extensible, especially for error handling. Nice work!
I personally use it for video media as well. One hit on my keyboard and the RSS item opens up mpv with the video playing. Computers are built to parse text, no reason to do it by hand sifting through websites with your mouse.
> DNS developers frequently see immense complexity not as a problem but as a welcome challenge to be overcome.
I think all developers favor complexity. Complex widgets are just more fun to make, they make you more money, and complex things are just way easier to make than simple things. If you have a problem, throwing more software at it is just what you do. And if you want to retain backwards compatibility, or not have to build around something, you usually have to add complexity.
I'll tell you a dirty little secret of the protocol design.
Say, you want to design a protocol with reliable delivery and/or loss detection. You will then have ACKs, send window and retransmissions. Guess what? If you don't follow windowing semantics of TCP, then one of two things will happen on saturated links - either TCP will end up with all the bandwidth or you will.
So - surprise! - you have no choice but to design a TCP clone.
[ EDIT ]
That said, there is a fundamental problem with TCP, when it's used for carrying secure connections. Since TCP acts as a pure transport protocol, it has no per-packet authentication and so any connection can be trivially DoS'd with a single fake FIN or RST packet. There are ways to solve this, e.g. by reversing the TCP and security layers and running TCP over ESP-over-UDP or TLS-over-UDP (OpenVPN protocol). This requires either writing a user-space TCP library or doing some nasty tunneling at the kernel level, but even as cumbersome as this is, it's still not a reason enough to re-invent the wheel. Also, if you want compression, it's readily available in TLS or as a part of IPsec stack (IPcomp). If you want FEC, same thing - just add a custom transform to TLS and let the client and server negotiate if to use it or not.
I mean, every network programmer invents a protocol or two in his lifetime. It's like a right of passage and it's really not a big deal. Unless it ends up attached to a brand name, in which case it starts attracting snarky remarks like this one :)
You won't see a correction. What you're seeing is the world change before your eyes.
Feel free to refer back to this comment in a year or two.
E-commerce is sitting at 15-20% of all retail, it will almost certainly go to 50% this decade if not higher - it's inevitable.
Value is more than revenue or profit and P/E ratios largely don't matter and haven't for a long time. Competitive positioning, long term leverage, and other things that you can't easily price matter just as much. This is about total and utter dominance in a sector - just look at the investments Shopify is making in delivering capital to small businesses or building out a fulfillment pipeline.
Amazon's P/E ratio has been at or over 100 for the last 20 years. It was at 300 in 2012 (would you have bought then?). If you looked at that and that was the basis for not investing, you are using outdated views to invest. These companies are different and it's worth figuring out why.
If it was up to me I'd never "upgrade" beyond Windows 7. Same thing with OS X, I'd like to stick with the version I have pretty much indefinitely. Not only OSes but most software in general are adding more and more features I don't care about & don't want, while adding onerous, user-hostile patterns that detract from both my freedom and the convenience that I usually expect from the software I use.
Frankly, as time goes by I'm finding it harder and harder to keep using Windows/Mac as my primary computing OS because every new update adds things like forced (or extremely-hard-to-avoid) updates, nagware, analytics I don't agree with and is difficult to remove, more-prevalent DRM, UX regressions (IMO) and "change for the sake of change" that provides no measurable or perceivable benefit to the user. I could honestly keep going on and on about this.
It's like software companies never heard the idiom "don't fix what isn't broken", and they are interminably obsessed with reinventing the wheel on a biannual basis.
Re-reading what I wrote, it's interesting & telling that I started with "if it was up to me", and that's really the problem: it's basically NOT up to me what I run on my computer. Want to play the latest award-winning game that is revolutionary, life-changingly awesome? Hope you have absolute latest [whatever] OS with all the updates and whatever other stuff that is required to run it (anti-cheat spyware is my favorite). Heck, how about stuff required for your business, non-optional software (like Zoom)? You're just forced into it. If this stuff you're forced into has problems like, say, wiping your entire hard drive because of a bug in a forced Windows update?? Too bad I guess? That's what you get for using a computer? There are so many glaring problems with this situation, it boggles my mind that people seem completely fine with it.
