It looks like Konform is a nice choice. Anyone tried it?
post
I'd really love to see some Blink based browser comparisons, especially Trivalent (which is supposed to be compareble to Vanadium).
And also Dillo and Netsurf.
What are you curious about with Dillo And Netsurf? Isn't it safe to assume at this point they will both be 0 across the board for all the queries in the report?
I think we need a different testing protocol for them to be interesting to include. AFAIK they don't have add-ons that could be interesting to test either? Do you have any suggestion for step(s) you think could be added to the test in order to make those meaningful to include? Or is my assumption about Dillo and Netsurf out of date?
Oh, thanks for reminding me of Trivalent, I realize now I've come across it before but totally slipped my mind. If/when testing for chromium in place I think this can be interesting to sample next.
Honestly, I'm interested to see if it can make it past my security. It's a good time to try. I might reinstall a new OS due to a pernicious Arch network issue and a DNS leak that connected me to a fucking Google server recently. Taking risk before wiping us better than after installing.
if you remove mullvads vpn app what do you see?
Assuming you mean the Mullvad extension (which is installed by default in MB) and not the Mullvad VPN app (which also exists but never came close to these machines) :)
That will indeed likely make a difference on Mullvad Browser numbers. However for now I'm not changing the "keep addons at stock defaults" invariant or the test matrix might get really out of hand.. Should we disable uBlock Origin in LibreWolf? How about uBO or NoSccript in Mullvad then? Konform Browser loads uBO but only if its apt package is installed; should we do that? What happens when we try to explicitly opt out of everything under Preferences in Firefox? I guess the last one is something to actually consider but for now not touching the addons.
(Would be super cool if anyone else tries this out and reports back though! The compose should hopefully be straight forward and easy to get started with if you are on Linux and have podman available. The report mentions it TL;DR we had to work around the oBO install in LW not properly utilizing the proxy (?) like this and I think same approach could be used to Uninstall Mullvad extension from Mullvad Browser and prevent it from even loading)
since others have mentioned others to try I am considering waterfox vs librewolf right now
In case you want to try this for yourself, adding container and running test for Waterfox should be about same as for Floorp that I wrote about here. Then you can really see what's going on and reason about the difference when you see the URLs and stuff.
BTW the purpose of the report section here isn't "look at my numbers and take my word for it" but "here's some examples of things we can look at with this". Please keep in mind both the Limitations section and that it's intended as showing one way to easily and independently compare browsers yourself. Just reproducing the examples shown and then scrolling through the .har files JSON is a great start. Of course, me and I assume others would be very happy if you want to share anything that comes out of that so that we can bring people up together. I'm sure there's a lot more useful insights to derive even with a small and scoped testing protocol like the one in article and wouldn't mind input of any nuggets other people come up with :)
So essentially, Mullvad is the only one out of the browsers tested that doesn’t leak notable amounts of data on first launch.
At least in most cases, the data is being leaked back to the developer and not third parties.
I don't think the data supports that. I'm curious what makes you single it out. Mullvad is in the top-tier but it is not alone (or clearly #1 - like the post gets into - it gets nuanced and I think any attempt at general objective "top 5 ranking" will be reductive to the point of being misleading or plain wrong. So I'm not trying that here). Read again? :)
For example of nuance displayed in results:
### Number of requests
119 firefox
81 firefox-esr
0 konform
7 librewolf
30 mullvad-browser
62 zen-browser
You’re right—they’re all doing differently privacy impacting things, but there are no “winners”.
There can still be winners, the good, the bad, and the ugly. It's just that we have to engage a bit deeper than a quick scroll and a oneliner to figure it out^1^ than that.
they’re all doing differently privacy impacting things, but there are no “winners”.
The difference matters. Looking into the raw URLs and bodies involved is enlightening. Apart from that, which other queries can we run with jq (or other tools) can we add to the post to add more useful dimensions?
^1^: The answer might be different for each of us and depend on what we're doing at the moment. Different situations might call for different browsers.
At least in most cases, the data is being leaked back to the developer and not third parties.
What is this based on? Why not see if that assumption is true^1^? There's quite a big difference in nature and quality here between them. This doesn't really come through in the data aggregation put on display in the post but I hope more people will try to run this on their own. Zen and Mozilla are the only ones with significant (and it is significant) telemetry of their own at all between these while LibreWolf and Konform have 0 data going to the devs, for one.
The whole idea here is to be able to achieve more nuanced and accurate understanding so more educated decisions can be made and enlightening conversation be had. Not just keep rehashing the same memes we based on vibes and hearsay.
Was hoping more for answering questions or getting new input than shooting down uninformed takes 😅
^1^: Well, staying inside the system we can't prove that no sharing with third-parties is going on if we only see one domain involved. But that is not the case everywhere here. We can easily see when separate servers operated by multiple parties are involved by looking at the URLs and looking up the domain names. And then we can go look at what's being sent to where.
Thanks for sharing! A human-friendly summary of the gist of those results would be super helpful. This looks interesting, and you may have rekindled my interest in LibreWolf over BetterFox-hardened Firefox again, but I'm not entirely clear on what your results mean in human terms.
Thank you for kind words!
Ah, then the hope is that this curiosity will trigger you to dig into it yourself (for example using the provided tool or taking inspiration from it) so that it starts making sense! I know it's an unconventional format to refrain from laying out my own opinions and analysis but that's my thing today. So much "everyone knows" and vapid third-hand takes flying around these days that I think we would do well to actually verify (and pick up related knowledge in the process) rather than take forum comments and blog posts for gospel.
OK, all right, I can try. I guess I can point at one thing in the Mozilla telemetry at the very end, doesn't that look very fine-grained if you look at the URLs (addresses) listed?
We can tell that many of the actions I took were communicated to the mothership for analysis and product improvement. Is this data really anonymized (or anonymizable)? Is it a reasonable amount for a user that has not opted in? My professional and personal opinion is: It is not.
But! That's just one isolated example. And an extremely limited view. What about Zen? Chrome, Edge and Safari weren't included here at all. And it's not at all looking at what happens for a user who probably cares about this: when you go to settings and disable all the telemetry. See I just said that one thing about Mozilla Telemetry and now I'm going to have to run some new tests and write reports about them for days just to set that record straight!
Maybe I'm odd but I think it's many (100?) times easier and quicker to gain understanding of the kinds of stuff we're looking at here by getting hands-on than to communicate it verbally. And I'm concerned with this limited attention span so many people are afflicted with these days, and look at how long this comment is already, no we're done with me telling you how it is, let's wrap this one up and get on to the juicy stuff.
There's an expandable section Basic test environment usage under Testing procedure but I realize now that might be easy to miss...
Anyway, to start it: Install podman, docker-compose (v2) and MITM_BROWSER=firefox-esr podman compose up --build. That should be it.
Then the browser pops up (hopefully), you do your thing, and after you Ctrl+C in the console, it will quit and the proxy will dump the recorded .har file which contains all HTTP and websocket traffic that went through the proxy in cleartext, in JSON format. There're tools online that can help visualize I think but nothing I can recommend off the bat. Simply cating it to the terminal or opening it in a text editor can be educative. Also playing around with variations of the jq snippets and see if you can come up with questions of your own to answer. Or if anything in my numbers make you scratch your head or say "wait a minute" dig there.
In case you want to take a look at what the thing does before running it (trust me bro), these are the files involved when you run that compose up command:
- entrypoint
compose.yml - imported
compose/proxy.compose.ymlfor mitmproxy - The browser
Containerfile(aka Dockerfile)
This is really cool, looking forward to seeing floorp and brave
Thanks! Adding Floorp should be straightforward if you feel like tackling it yourself as it's "just another FF fork". Adding a new browser consists of adding a new Containerfile for it. I guess Floorp might be most similar to Mozilla firefox out of the existing ones. PRs much appreciated for new browsers as well as any interesting queries to get more insight into data I can run on existing dumps and add to Report section.
They have official PPA: https://ppa.floorp.app/
For Brave got it running but didn't yet figure out why it crashes as soon as I try to proceed with the onboarding. Judging by the probably unrelated error noise in the console, it might be trying something weird with a graphics driver or hardware sensor and not gracefully handling not having access to whatever it is 🤷 But didn't even ldd or strace it properly yet so maybe just a missing library.
There's a lot that could be done but had to wrap up and publish somewhere.
Hi, I'm new here, first time posting to this community, was hoping this could be well-received here.
I see this starting to attract downvotes - is this considered breaking any rule, are cross-posts frowned upon in general, is the content too basic for you 1337 h4xx0rz, title not serious enough, or some other issue with the post? Feedback appreciated.
all 12 comments