Changing permissions is not the same as bypassing permissions

If you're not that worried about storage then you can just make copies if necessary, then you don't really have to worry about permissions (apart from read, which is typically default for the same group). But yea if there's any chance more than 1 person might work off the same copy of data on HPC, make it read only for the peace of mind. Regarding conda envs, yea I have a few common read only conda environments so that scripts can be used by multiple users without the hassle of ensuring everyone has the same env. Quite useful.

Your job as sysadmin is to adhere to your organization's policy, no matter how stupid and hindering that policy might seem to you.

You're knowingly giving your users a workaround to their NDA, which puts all of your jobs and your data confidentiality at risk.

You've got no business with root privileges.

.. don't they? usually, owner of the directory can change its permissions, and ACLs when those are enabled for the mount

Is there an application that allows easily sharing files on one Linux system? That would be nice!

this sentence made me remember this: https://github.com/9001/copyparty

You are in way over your head

Stop now before you get yourself in hot water

That is not how permissions work

Can you check back in here and let us know if it worked?

If it's a compliance problem, I get it. From a practical standpoint, FTP or WebDAV don't require installing anything.

You can set acls on directories that get applied recursively. This makes ist possible to have all files be the correct permission. I am on the go right now but you should look into setfacl. It's been a while but I am pretty sure that worked. That way you should even be able to say which groups or users can do what with granularity.

It is not something I can setup on that server, I would need a separate server to set up something of that kind.

You don't need additional storage. It's one program you need to set up.

This is a good idea and something I may setup once we setup our own compute server. However at that point wouldn't a synced directory be a better fit for the purpose? Such as you define a directory on the external server to be used to share data and every user syncs it to their own share on the main server to get all the shared data through rsync or unison.

Just throwing it out there, I'm not sure if that fits your use case.

Thanks, this is a great idea! I can see you have been doing this for a long time and you're talking from experience. Regarding shared data: I use this more as a way to give raw data to other people and collect results from them. I use it mostly as a temporary directory used to transfer data, anything significant will get copied over to my share and backed up.

I can see how you could be worried about storage quota, luckily I don't have that many people to worry about. But it is funny you mention it as I could really see someone stashing a few conda environments in there just because they finished their inside quota...

I see! Well, I currently do not have another server that has so much storage that we could use for thi purpose. Maybe in the future and that will solve a bunch of problems, this is only one of them.

We do have a storage server, but that is local only and backup only: not going to open it to the internet.

It is indeed a solution. What is absurd to me is to have to consider such a solution that requires two servers.

I don't want to override user permissions, I want it to be easy that users can agree to a directory being shared and in such directory group permissions get overridden by some laxer restrictions through explicit authorization of the user. I'd like for this to exist. This would not make permissions useless, just allow for an easy way to share files across users on a filesystem.

I am not bypassing any policy: the HPC Is there to collaborate on and data can be shared. Not having a shared group is not a policy, it's just that not all users are in the same group and users are added to just one group by default. We are indeed allowed to share files, hell most of the people I want to share stuff with are part of my own research group. ACL is allowed on the HPC. I'm asking how to properly use ACL.

If you have anything actually useful go ahead, otherwise don't worry that I know better than you do what I should or should not do.

Thank you for the reply and for trying to help out. You may have misunderstood the question: don't worry it is ok.

It sounds like you are out of your element here which is risky.

I am ok, it is not risky do not worry. Judging from your answer I'd say I have a bit more experience than you do.

I don't know what your background is, this is mostly hindrances when doing research. Administration has taken over and demands deciding how research should be conducted without having any idea on how it should be conducted.

You may see it as a red flag, myself I have this very clear that I do not want to follow their bullshit. If I'm losing my job so be It, However you may be misunderstanding: I am not going to lose my job over this, nobody is getting hurt, and I am sharing a bureaucratic process that is fairly common over here in public institutions. This is not some large corporation that has to keep secret the time schedule of it's workers: if you wish to come over I have full liberty of deciding to show you anything I have on my computer and most projects I am working on. Yes, there's a couple things I can not show you, but everything else is my own job and up to me to show to whoever I wish. Institutions may retain part of the IP if we decide to commercialise stuff, but I am the author and I am free to share anything I don't have an NDA on.

I am afraid you come from a very different background and you are misunderstanding my situation.

I'm in a similar position as you. Our lab has a partition on HPC but i need a way to quasi-administrate other lab members without truly having root access. What I found works is to have a shared bashrc script (which also contains useful common aliases and env variables) and get all your users to source it (in their own bashrc files). Set the umask within the shared bashrc file. Set certain folders to read only (for common references, e.g. genomes) if you don't want people messing with shares resources. However, I've found that it's only worth trying to admin shared resources and large datasets, otherwise let everyone junk their home folder with their own analyses. If the home folder is size limited, create a user's folder in the scratch partition and let people store their junk there however they want. Just routinely check that nobody is abusing your storage quota.

EDIT: absolutely under no circumstances give people write access to raw shared data on hpc. I guarantee some idiot will edit it and mess it up for everyone. If people need to rename files they can learn how to symlink them.

I think the current code I see being generated is generally “good enough”. I’m not comparing it to perfect: I’m comparing it to people.

If this were true, then open source projects would have much less of an issue with pull requests from sloperators.

Have you seen output in which satirical code is actually included?

I wouldn't expect to see it. Satirical code requires more thought than an LLM is capable of putting into its writing - you need to understand what is expected of whoever you're satirizing, and then you have to take that expectation and take it a step further into the absurd. Without having that context of something that is specifically being satirized, what you have instead is just incorrect code. And again, the LLM is incapable of valuing proper code over intentionally wrong code, so it's going to poison the database to some extent.

And LLMs don't drop big chunks of copy-pasted code from Stack Exchange like an intern would. They work one token at a time. (Which is why trying to get them to understand that quotations need to be all in one piece is a futile endeavor.)

Besides, 'satirical code' is just one example of the many things that can poison the training. I couldn't even begin to enumerate all the things that could mess with it, and honestly I'm surprised that LLMs do as well as they do considering they likely have all sorts of cross-language screwball connections (which may be why it has such a tendency to make up libraries; it doesn't necessarily understand that a common PHP library doesn't exist in Java).

do you not believe that either (a) these types of trivial issues would be caught by a person whose job was just to audit output or even (b) this type of issue could be caught by specially trained domain limited AIs designed to check output?

These issues could be caught by someone whose job it is to audit code, sure. The problem is that sloperators often don't audit their own stuff well enough. They leave it to the open source repo's admins. When pull requests from overeager noobs were infrequent, it wasn't the problem; they could gently correct them, the repo would stay high-quality, the noob would learn, everyone is fine. But now, sloperators are dumping low-quality pull requests on the repos faster than the admins can sort through them - because it now takes less time to produce slop code than it takes to determine whether or not the slop is worth including. The admins are swamped, because they can't sort the wheat from the chaff fast enough.

A domain-limited AI designed to check output would be useful - if it could be trusted. Open-source project admins are some of the best coders out there, and they vastly outstrip the capabilities of LLMs. You're suggesting that we replace THEM with an agent. They are in that position because they're right far more often than they're wrong when it comes to understanding the code as it exists, and how incoming code would impact it - or at least they're right often enough to keep the project alive. LLMs will be worse at that job, I guarantee it. They'd be fast, but they'd be wrong too often. This is the primary issue with LLM agents.

I recommended Self-hosting whatever middleman software. Opencloud, WebDAV, S3, FTP, anything you put in the middle can accomplish what you want.

Yes. That's what I recommended. Self-host whatever middleman software. Opencloud, WebDAV, S3, FTP, anything he puts in the middle can accomplish what he wants.

Someone has clearly never created a temporary fix. That shit is eternal.