I just finished augmenting some of the Spider and WikiSQL data on huggingface [1] I initially intended to train a text-to-sql LLM that would take a natural language question and be provided the CREATE TABLE statements to provide some grounding for the response. So instead of hallucinating the column and table names by using the question alone, I was hoping the CREATE TABLE statement would limit the choices. We'll see if it's useful or not, but funny enough I came across this article after I finished the dataset.[2]
I'd definitely like to see how others are doing it.
I've got it running on Chrome v113 beta on Ubuntu with an older AMD RX 580. The feature flags don't seem to be taking for me in chrome GUI but if you start chrome from terminal like this it works.
I work as a data engineer for a Defense company. It's not always stable but depends on the contract you can get on, some have a pretty broad scope and arent going anywhere, others are small projects with less stable funding.
Most of what I work with is Databricks, Python, Pyspark, etc. So most coding is done in notebooks. I haven't felt that there was a lot of restrictions coding wise, but getting access to different databases, clusters, and especially on boarding can be a pain.
Environment wise its been pretty good, you find even the clients are often contractors themselves, but if you dislike them you can always roll off and onto another project, it seems everyone needs more devs
There's a few resources out there for smaller companies to try to get into defense, mainly networks and incubators. [1]
But you'll also see SBIR/STTR funding topics (program run by the Small Business Administration) [2]
You can also try checking out some of the bigger Defense contractors, some have incubator programs and are looking to expand their small business ecosystem for subcontracts, part of that includes funding. (Disclaimer, I work here) [3]
I've used this recently to do some fuzzy matching of column names in datasets, I also added it to a small python one-liner library I've been making for practice. p.s. don't give me flack, I know this isn't an efficient way to do things.
jaccard = lambda A, B: len(set(A).intersection(set(B))) / len(set(A).union(set(B)))
If I'm not trying to build a very specific graph or chart, and just exploring data I usually use either Rawgraphs or Sqliteviz. Rawgraphs is nice if you just want to swap visualizations out with smaller data as is, sqliteviz seems to handle much larger datasets and let's you use SQL if you want to change the resultset. Both seem to keep data local too and I know sqliteviz works offline, rawgraphs might too.
Very cool benchmark. I don't think this will be a huge difference but could you try mypyc? I'm definitely curious how it compares. It uses mypy to perform type checks and claims "Existing code with type annotations is often 1.5x to 5x faster when compiled. Code tuned for mypyc can be 5x to 10x faster."[1]
This seems opposite to what their new president said last month:
"From the beginning, the team behind Signal put people and their needs at the core of their commitments. They understood that iron-clad security is fairly pointless if people can’t use, access, or feel comfortable with it. In other words, if my friends won’t use a messaging app, it doesn’t work as a messaging app. It works as a thought experiment, at best. Understanding this, Signal’s developers and designers created an app that honors people’s needs and expectations, while maintaining strict privacy promises."[1]
I'll echo the other comments here talking about network effects, onboarding friction, and social capital wasted convincing friends/family.
I'd definitely like to see how others are doing it.
[1] https://huggingface.co/datasets/b-mc2/sql-create-context
[2] https://blog.langchain.dev/llms-and-sql/