[–] 3 points 9 months ago* (last edited 3 months ago)

dupeGuru has a great GUI for finding similar looking images (videos are not supported). See How to deduplicate files on Linux with dupeGuru for a tutorial. There's a similar tool for videos called videoduplicatefinder but I did not try it out.

I will now share my experiences with using dupeGuru. I ran it in Picture mode and selected "Match pictures of different dimensions". It was nice to see that it found a few low-quality JPEGs that look identical to higher-quality PNGs. One feature that I really like is the option to "ignore duplicates hardlinking to the same file"; let me explain why: I sometimes download the same image into multiple subfolders of my fan art collection, so I want to keep both images despite them being exact duplicates. This can be accomplished by running rdfind -makehardlinks true . to turn exact duplicates into hardlinks, and running dupeGuru afterwards.

I experimented a bit with with "Filter Hardness". On the first run, I set it to "≥95% match" and the results did not contain any false positives! I later went down to "≥70% match", which reported more duplicates, but also some false positives. If you have many comic doodles in your collection, choose at least 80%. However, if you want to detect crops of other images, you have to choose a lower percentage. On my collection of 3000 images, dupeGuru took about 5 minutes.

dupeGuru has one major downside: The current release version ignores webp images, but you can build it from source (Arch users can install dupeguru-git from the AUR). Another downside is that it automatically chooses the biggest file among all duplicates as the reference file. This is often a good heuristic but fails badly with webp images that have better compression ratio than other formats. You can fix this by going to Actions → Re-Prioritize Results and putting "Dimension: Highest" at the top of the list.

  • source
  • [–] [S] 1 point 11 months ago

    Yes, though something that can render in 3D would be nice for visualizing what the data mean. And while that works for 3D, suppose you need something with a fourth axis. You would have to add another level of tabs to keep your tabs in for each new dimension. Not that you'd usually need more than like 5D, but still.

  • source
  • parent
  • context
  • [–] 4 points 11 months ago*

    You might want to look into GNU Octave, it's like an open source version of Matlab which is often used for managing high dimensional data.

    https://wikipedia.org/wiki/GNU_Octave

    A couple other approach depending on the data and use are using relational database, storing the data in a database normalized form, then using a declarative style language like SQL to combine or exclude data in operations. And if it's mostly physics spacetime coordinates xyz+t tensor frameworks and libraries are built on top of most popular general purpose programming languages (python, c, java, etc) for that purpose.

  • source
  • [–] 1 point 11 months ago* (last edited 11 months ago)

    Hmm. I see. I'm not aware of something that I think would be particularly good for that which already exists, but it's the kind of thing that sounds like it might be possible to hack together quickly with an HTML table, contenteditable (or input tags), a few buttons/select tags/etc, a bit of JavaScript, and maybe python3 -m http.server or a Python script using tornado depending on what was needed exactly... but maybe I've been in web programming land too long.

  • source
  • parent
  • context
  • [–] [S] 3 points 11 months ago (1 child)

    Minecraft is not a serious solution. I could get it to work but it would be terribly uncomfortable to read and edit, almost certainly worse than layered 2D spreadsheets.

    Currently i'm making a constructed language and trying to work out what all the possible syllables are given the fairly simple syllable structure. There are tools specifically for this, but there have been other times when a 3D spreadsheet would have been very helpful even if i can't still remember what problem i had.

    Another example, also for constructed languages, is tables of conjugations and declensions in fusional languages. That is, a language where a single prefix or suffix indicates a specific case, person, number, etc. and there are a bunch of these that need to be mapped out.

  • source
  • parent
  • context
  • [–] 5 points 11 months ago* (2 children)

    You can use multiple sheets in a workbook or multiple layers in an image editor to get a third dimension in existing 2D data editors. I've done some weird things like pack numeric data into RGB pixel values when I needed to annotate an image with a couple bytes of data per pixel in irregularly shaped regions...

    If Minecraft is an actually serious possible solution to what you're doing, maybe a world editor for it would be useful? I don't know what's popular now (been OOTL for too many years) but people used to make custom maps in MCEdit; some of the 3D brush tools and such might be useful depending on what you're doing?

    Can you provide more details on what you're trying to do?

  • source
  • [–] 2 points 1 year ago

    Glad that helped the decision.

    As someone who does this for privacy, I just end up using a password database to make notes on each provider. It holds the relevant authentication, payment, address, etc data in the notes. I religiously update it when I create something new or have to change. Categories make tracking easier, I.e. payment method = category, same for addresses, etc.

  • source
  • parent
  • context
  • [–] 3 points 1 year ago

    dot is the file for.at used by graphviz, and tbh it seems pretty close to what you're describing. There's a number of tools that can consume (and produce) the file format.

    It's the format I've worked with on occasion (though admittedly it was because it was what was produced by other tools).

  • source
  • [–] 2 points 1 year ago

    Awesome!

    For what it's worth, I got "fancy" with graphviz a while ago. One of the things it can output is SVGs, which can be used in a web browser for interactivity. For example, I set up one graph to highlight all the branches that are connected to a selected node, and I put links to navigate between different graphs. So there's quite a bit possible there.

  • source
  • parent
  • context
  • [–] [S] 1 point 1 year ago (1 child)

    Thank you! I do not have any experience with graph databases or RDF, so you've introduced me to a very fun rabbit hole :)

    On reflection, I think the route I was going down was overkill. I only need to graph small numbers of dependencies (tens or hundreds of objects). Graphviz is probably still best for my use case, moving onto a graph database or similar if I ended up needing to represent thousands of very complex dependencies.

    The main motivation for this was wanting to keep track of all my accounts and the information they have about me in a structured way. I made the mistake of using a few different email providers and phone numbers and trying to segregate everything. That wasn't worth it and I want to move all of my accounts back to pointing to a single email and phone number. But, I never made complete records of which accounts I have, so I wanted to come up with a good way of documenting that, so that I can be sure when I can stop monitoring / delete old accounts.

  • source
  • parent
  • context
  • [–] [S] 2 points 1 year ago (1 child)

    What other types of outputs are you looking for?

    I think the main outputs I'd like are:

    • Given a "thing", show me what is dependent on that thing (find-dependers email1 - printing bank, credit-card-company)
    • Given a service provider, show me what things it is dependent on (find-dependencies electricity-provider - printing address, email2)
    • Graph the dependencies of a thing or service provider (graph-dependencies email1 - outputting a graph with bank and credit-card-company pointing to it)

    My second thought would be some sort of database software.

    I agree, a database would be a good fit for these relationships. A bit of investment up front, but the schema should be very simple.

    Graphviz would have been my first suggestion.

    My main worry with graphviz was that I thought I would be stuck with a big graph of everything and unable to filter.

    But, now I've found gvpr, and I can see that arbitrarily complex filtering is supported.

    I'm thinking for this use case at least, I should be able to build a big dot file and then have some gvpr scripts that generate graphs just for the dependencies I'm interested in. I should be able to hook into gvpr to do the terminal output too.

    So, long story short, I think I'll give graphviz a go, and if that fails, a database with some scripts to generate the dot files on the fly. Thank you!

  • source
  • parent
  • context
  • [–] 2 points 1 year ago (2 children)

    Graphviz would have been my first suggestion. What other types of outputs are you looking for?

    My second thought would be some sort of database software. LibreOffice has a MS Access equivalent (Base) where you could keep a table for each type of data, then express relationships. If you're already comfortable with SQL it might work well, but otherwise might be overkill (though you could get just about any output you'd like from it).

  • source
  • view more: next ›