How to store large amounts of data in JD while syncing with other devices?

J.D Concepts 11 Core concepts
1

Hi all,

I’m currently getting started with the Johnny Decimal system and love the idea of keeping my life organized with it. But I’ve run into a practical problem when it comes to large amounts of data and syncing across devices.

By “large amounts” I mean:

  • Tons of smaller files (e.g. coding projects)

  • Some quite large files (videos, high-res images)

  • Or both: tons of large files

The setup:
I want as much of my JD system as possible synced between my laptops, phone, and tablet. For syncing I’m using Syncthing. The index is a no-brainer — small, light, easy to keep everywhere.

The trouble starts with things like:

  • Photos from trips (which I love to store with itineraries, bookings, etc. in their JD context, like 15.4x or 15.5x)

  • Coding projects

  • Video footage and other big stuff

Keeping files in-context is important to me. If my holiday pictures live in 15.4x, they’re right next to all the related trip info, which helps my melty brain to remember details more easily when looking things up later.

But here’s the problem:
If my JD “root data” folder is synced across devices, I have to:

  1. Sync everything (impossible — some devices don’t have the space)

  2. Create tons ignore rules (which feels brittle and messy)

Alternative approach I’ve thought about:
Split big categories into separate “root folders” outside my main JD structure — e.g.:

  • Coding projects

  • Pictures & videos

  • Everything else

But… that feels wrong to me. I really like having everything neatly inside my two JD root folders: index and data. Splitting things out feels cluttered and like giving up some of JD’s elegance.

If Syncthing had “sync on demand” (like OneDrive or iCloud), this would be easier. But it doesn’t and neither have a lot of other sync tools (especially self-hosted).

So my question is:
:right_arrow: How do you handle large data in JD when syncing across devices?
:right_arrow: Is there a clean way to keep files in-context without either bloating every device or creating a mess of ignores?

I saw some comments in Syncthing configuration -- (selective) syncing to many devices , but nothing of more principal nature.

I’d love to hear how you all approach this!

PS: I hope this fits in here. I thought about posting in J.D Concepts > 31 Apps and utilities , but I guess this is not about a specific tool. Might as well substitute Syncthing with other solutions and have the same problem. That being said: feel free to move this, if I misjudged.

2

Any reason you’re not using git for the coding side of things? That would at least solve a part of this problem.

3

Create a wildcard rule that matches (not synchronised) or your preferred indication?

FWIW I keep all of my dev-project stuff in ~/dev, outside my normal ~/Documents. Because it’s already synchronised up to GitHub and I don’t need Syncthing chewing on 200,000 files every time I npm i.

4

Hi there @mh166,

yup, this nut is still uncracked. The thread you linked to is the state-of-the-art, which is, as you point out, fairly lacking.

I ran into this again on Saturday too, as I’m starting to deal with my multi-decade stash of photos scattered in various degrees of duplication across myriad hard drives. Like you, I want to have series of photographs right there with the events they relate to. Actually, I do plan to keep them in a separate data directory but well-named so I can select based on event names; but the problem is that this data directory will already be several 100s of GB and I plan on adding to it indefinitely in the future!

I have resolved to try git-annex again. It is the only free and open source tool I know that is designed exactly for this use case, and actually does a pretty good job of it … it’s just that it has such an infuriatingly confusing UI. That’s because it’s based on git, and operates in the same general UI philosophy, with myriad flags for everything, and a confusing(ly documented) data model. Just like with git, it’s surprisingly easy to get into irrecoverable situations.

However, while it was working it solved this problem of selective syncing perfectly. And maybe things have improved a little in the intervening years … So I’ve decided to try it out more carefully, with all my data 300% backed up at every step. git-annex is not a backup solution.

In the commercial sphere, I know of Resilio which I’ve never used and don’t know much about.

As I understand it, this would work for one-time configuration, but not if you want to change what is where in the future. Syncthing won’t be able to handle the case where something was synced, and then add it to the ignore list. It’s not going to delete the copy it already has.

5

I’m going to sound like a broken record but a home server is what you need. Old computers with enough hard drive space for all that data work wonders.

6

I used Resilio pre-Syncthing. It worked for a while but my feeling is that they’re much, much more interested in business customers, and that the consumer side felt like an afterthought.

7

This is the answer, yes. It’s what I do, in my case with a base-model Mac mini (bought refurbished from Apple), and 16TB of attached storage.

8

I deleted a paragraph from my post containing my thoughts upon opening Resilio’s website yesterday … It’s good you point out the (large) business focus. The case-studies and pictures indeed suggest business use. So I’ll give them the benefit of the doubt, because while their recent move to a model similar to Obsidian’s, with the service free to use, in some ways sounds encouraging, there’s something missing in the site’s language that makes me feel like care is not being taken.

9

well, I would say yes and no. Network access can work fine for some things. For example, for my image collection, I think ultimately it needs to be a web app. But sometimes, you need a lot of your data locally. Either for latency or security concerns. The issue is how to selectively sync parts of your huge collection. Do I manually copy my huge video files to my laptop for editing, and then manually upload the results? Or is there a system that can automate the copying and the keeping track of what is where?

10

I’ve got a QNAP NAS this year, my single point of truth.

My multimedia is in eagle.cool which creates it’s own database. Actually I have been busy and lazy so never got around to sorting them.

This lives on the NAS and I can access it via a windows 11 VM, which is too slow. If my NAS had a i5 cpu it would be doable. I only use the VM for downloading and seeding torrents. (*Rr apps and qbitorrent)

I access my content at home PC so not needed on the road. But I can see that logging into a VM remotely is your best solution, but SSDs for latency if editing.

In two years I will probably upgrade to some new NAS that has a RAG and LLM and it can auto-tag and I can verbalise to sort and access files, lol.