Very interesting, thanks for the write-up!

Just as an aside, but you also noted it yourself

I thought PNG was inherently lossless but the pngquant tool proves there is a lossy compression algo.

PNG itself is lossless, it's just that pngquant "manipulates" the image to be better compressible.

And I didn't know about DjVu at all, that's really good to know!

Thanks! I did not realize there were different compression levels. So I discovered optipng. If I scan to PBM, that can be fed into optipng set for max compression:

$ convert sample_scan.pbm sample_IM.png; # ImageMagick for comparison
$ optipng -out sample_optipng_default.png sample_scan.pbm; # run with defaults (moderate effort, -o2)
$ optipng --strip all -o7 -out sample_optipng_min.png sample_scan.pbm; # max effort (smallest result)
$ pdfjam --a4paper -o sample_optipng_min.pdf sample_optipng_min.png
$ cjb2 -lossless sample_scan.pbm sample_lossless.djvu; # DjVu for comparison
$ du -h …
24K     sample_lossless.djvu
68K     sample_optipng_min.png
68K     sample_optipng_default.png
72K     sample_optipng_min.pdf
80K     sample_IM.png

The tightest PNG compression w/metadata stripped (or even just the optipng default) is much smaller than ImageMagick’s output. The PDF produced by pdfjam is bigger than the png it contains (rightfully so). So I’m pleased with this. I’ll replace ImageMagic with optipng. And if a scanned doc is just 1 page, I won’t even bother with pdfjam and just keep the png.

I thought PNG was inherently lossless but the pngquant tool proves there is a lossy compression algo. Although I suppose anything can be lossy with image processing done independently. I did not fiddle with pngquant because it seems to only be relevant to color reduction, which makes no sense on a bitonal image.

DjVu is a clear winner. ⅓ the size and still lossless. But there is no LaTeX pkg for DjVu so the smaller size obviously comes with a big trade off. But then we must consider that a DjVu file can be directly converted to PDF (LaTeX importable) using ddjvu. So we have:

$ ddjvu -format=pdf -quality=150 sample_lossless.djvu sample_ddjvu_lossless.pdf; # 150→highest possible quality
$ pdfimages -all sample_ddjvu_lossless.pdf innards; # because we want to see what’s inside that thing
$ du -h …
24K     sample_lossless.djvu
44K     sample_ddjvu_lossless.pdf
44K     innards-000.ccitt
4.0K    innards-000.params

$ cat innards-000.params; # to get an idea of what the CCITT is (fax or whatever)
-4 -P -X 2464 -B -M

The PDF size (from DjVu) is still smaller than the smallest PNG. And we can see that ddjvu did not do anything foolish like stuff a jpg in the PDF. It’s not fun to work with CCITT files, but I suppose that would be rare. In the end, DjVu seems like a winner for archiving because ddjvu can always make a PDF that is still smaller than a PNG, and LaTeX can work with that.

I don't know about compression on the PDF side, but PNG uses deflate compression, which, according to https://www.libpng.org/pub/png/book/chapter09.html#png.ch09.div.2 is "not the best-compressing algorithm known" (that's taken slightly out of context, it definitely is not bad). Also according to that chapter, most PNG implementations use zlib, which supports different compression levels. So the PDF container might have a way of compressing it slightly better, considering that 97/105 is not that good of a compression ratio.

You could try opening the image with an image editor that exposes the different compression levels (GIMP and Krita come to mind) and see, whether you can compress it further

I started to create a package. Rough prototype is posted here:

https://lemmy.sdf.org/post/60831598

Worked for me

Cool, but it gets inaccurate text to speech voices... It list all available for download, which is all possibly. Only Eng US is actually installed.

It thinks WebRTC is leaking stuff, but it's just sharing the VPN public IP, as it should.

And the top of the screen bounces as new lines of text are added, making it impossible to follow along until it's done adding shit... Then I scrolled up and worked back down.

Horribly annoying that is! Not surprised if people don't read through it all, or just leave immediately.

That's what you're hung up on?

see the other comments. pdflatex crashes when fancyqr is overworked, which I’ve seen happen when exceeding just 2 qr codes. There is also a typesetting problem w/big codes.

Even when it doesn’t crash the compiler, it’s very slow. None of the QR pkgs are sophisticated enough to store the computed result and reuse it from one compilation to another.

Thanks for the comprehensive answer!

And that's related to TeX how exactly?

I both like and hate that the blog post was printed out as-is and hung up for display.

You would care in some situations. Someone needs codes to be scannable from a long distance: https://hachyderm.io/@dolanor/117079618531657267

I sometimes produce a document that is littered with codes. If the qrcodes are small enough, they can be in the doc close to where they are referenced. The bigger they are, the more intrusive they are to typeset around. If you are doing a bilingual doc with a different lang on either side of the QR code, It’s harder to manage a big code aesthetically. Then we are forced to consider moving all the QR codes to the last page in a references section -- which is a trade-off because it means the reader must track down the right code for the reference they want to follow.

QR encoding is computationally intensive. When I use the fancyqr package (which uses colors), I can only get away with 2 QR codes per document or latex crashes. (edit: not really a crash but probably terminates with → “ERROR: TeX capacity exceeded, sorry [input stack size=10000]”).

Consider artwork as well. A stained glass worker really needs to keep the labor of cutting glass to a minimum and has a strong incentive to have a reduced number of bits. But then it’d be a shame to do a lot of labor to get a stained glass QR code that yields all uppercase text.

So the poor decision to use all uppercase for the smaller charset impacts in several unpredictable ways. That is, you cannot predict all the circumstances that small QR codes are needed. But they could have predicated that all uppercase is less useful than all lowercase.

what's the benefit of this? why should i care how big my qr code is?

Without knowing a dedicated app, this is how I think I would go about it:

1. Batch scan with a feeder into individual documents
2. Run all the images through magick -trim probably with some fuzz option as well
3. Might as well run some OCR on all the documents as well to embed the text ocrmypdf
4. I'd imagine for receipts you'd want to tag or label them, so do that as well, which seems like it would have to be a little hands on. Maybe make up a naming scheme, Date-Expense Category-Location-Notes (whatever works)
5. Sort by width, throw them in a Latex document resizing to fit 2-column width using minipage or another option, then maybe 1-column above a certain size.

It doesn't seem like it would take much more than a few hours at most to script up something that could do that. You could probably easily add a table of contents with the titles or info from the images (from step 4) as well to make it organized. Once set up, it shouldn't take more than a couple of minutes to go through a month or more of receipts. File the receipts by month (or quarter or year), update the document monthly (or however often needed), and you'd be pretty well organized.

Steps 1-3 are basically one-liners. Step 4 would be a couple lines of bash, and step 5 would be a handful of lines of bash. Really it's not much.

I think it should be feasible, if you can reach them. I’m done with github, who forced 2FA on my acct via email confirmation before my email address died. I’m not going to dance for Microsoft in order to give my attention and interaction to MS users.

In principle, I think if someone were to set up a TeX-specific forgejo/gitea instance, it would be attractive for LaTeX devs because they probably are not getting much benefit from be centralised on Github anyway. It was likely an arbitrary choice for them. If there were a LaTeX-themed git community, it would be attractive.

next question is how can we convince these packages to move to open source forges like forego or gitea?

I harvested the CTAN DB to research this. The results:

   1390 github.com
     90 gitlab.com
     36 sourceforge.net
     31 codeberg.org
     20 tug.org
     18 framagit.org
     16 bitbucket.org
     10 git.gnu.org.ua
      8 gitlab.gutenberg-asso.fr
      8 gitee.com
      6 dickimaw-books.com
      5 puszcza.gnu.org.ua
      5 heptapod.host
      3 git.robertalessi.net
      3 git.framasoft.org
      2 lists.tug.org
      2 git.sr.ht
      2 gitlab.ti.bfh.ch
      2 gitlab.inria.fr
      2 gitlab.adullact.net
      2 gitea.com
      1 uds-datalab.github.io
      1 tufte-latex.github.io
      1 todo.sr.ht
      1 texnia.com
      1 svn.tuxfamily.org
      1 svn.gnu.org.ua
      1 svn.code.sf.net
      1 svenharder.github.io
      1 savannah.nongnu.org
      1 savannah.gnu.org
      1 repo.or.cz
      1 qa.parsilatex.com
      1 plmlab.math.cnrs.fr
      1 osda.ws
      1 latex-project.org
      1 humenda.github.io
      1 guitex.org
      1 git.umaneti.net
      1 git.savannah.nongnu.org
      1 git.savannah.gnu.org
      1 git.ortolo.eu
      1 git.linta.de
      1 gitlab.science.ru.nl
      1 gitlab.reutlingen-university.de
      1 gitlab.git.nrw
      1 gitlab.fi.muni.cz
      1 forge.apps.education.fr
      1 bastien-dumont.onmypc.net
      1 archiv.dante.de

The 3 most common forges for LaTeX projects are shitty centralised walled-gardens (MS Github, Gitlab, & Sourceforge). Only 174 projects out of 1690 are using open access free-world/decentralised forges (~10%).

It is disturbing indeed. It also means reporting bugs on ~90% of LaTeX pkgs requires dancing with an evil gatekeeper, licking boots, etc.

Oяցαηιɕ WiTh SoMe Z̶̡̨̩̘͕̟̅̅̇̈́̒́̐̿̇͆̉͒͝a̸̛͓͉͊̇̒́̈́͑͒̿̓͆̏͘l̴̢̘͍̯͎͍͚͖͓̤͍͂̔̓̒̂͋̑g̴̛̩̮̞̃̅o̷̧̥̲̫̠̝̭̱̰̞̠̾̏̒͜

Comic sans, obviously

times roman

Comic sans.

seems it might mean that. found some digestible info here: https://wiki.debian.org/CUPSDriverlessPrinting

Thanks. I’ll have a look at some of those approaches.

(edit) I used a feature in the KOMAscript pkg to produce circles that reach the edge of the paper. I also used one of the approaches in your link to create a frame at the point where the /expected/ boundary is, so that if the frame has any missing lines it would indicate where the specs may be wrong. But I must say I don’t trust LaTeX to produce an accurate frame because some lines are closer to the edge than others even though I asked for 4.2mm on all sides.

Sounds like a fun exercise. Quick Google found this stackex question, might point you in the right direction.

Otherwise the print shop may have a verification document/method that could be adapted to the self-service printers? Sounds frustrating that they can't/won't assist more.

The staff at the shop I use the most did not know. But the self-service printers are different than the printers used by the staff. They might know for the printers they use, which is naturally more costly.

Creating something with symbols going all the way to edge seems like a good idea. I would not want any spacing between the symbols though, so I guess it would be non-trivial code.