Why does JD have nested folders?

Now I know this may be sacrilege, but I have and use a JD system I love very much.

I find that the main advantages of a JD system is absolutely knowing where something might be, having a clear bird’s eye view of my entire system’s scope because I’ve thought myself about where to put everything, and being able to allocate resources or see blind spots based on having a JD system with empty holes or potentially missing folders/notes in an empty ID that I know should exist. However, all these advantages have come to me as part of creating the Index and decidedly not maintaining the folder structure.

It’s been discussed before lightly in terms of how maintaining a flat folder structure can help re-organize at the index level not at the file storage level, or how some people want flat file systems to they can tag folders to multiple IDs. An idea that gets a bit closer towards the philosophical endpoint of dividing life into folders, is doing it in such a way that mutually exclusive tools read and write from the different JD IDs to be able to maintain a single JD system across various tech and organizational platforms.

However, the question I want to ask is deeper. Why have nested folders at all? Instead of something like:

10-19 physical spaces
└ 11 home
╚ 11.01 living room
...
└ 12 office 
╚ 12.01 filing cabinet
...

Why not just have a file system that looks like:

11.01 living room
11.02 bedroom
...
12.01 filing cabinet
12.02 central library
...

The only disadvantage I can think of is that as IDs get added the click-through browsability based on muscle memory of the JD maintainer gets weaker.

But this argument has two main limitations:

  1. At the single JD category level this muscle memory issue may well exist anyway.
  2. Not every JD ID will have a folder on every computer or technological boundary it crosses.

However, doing away with the whole notion of nesting folders confers many advantages:

  1. In any system where one can see the flat folder structure it would be quite easy to see which IDs are missing.
  2. Teaching people to build a JD system would now involve just creating folders without creating IDs to layout the basic needs the system needs to fulfill. This can then lead to instruction about how to group things based on similarity, goal, physical proximity or any categorization the individual maintainer can think of.
  3. Reduces the number of keystrokes in every filesystem to reach a distal JD ID.
  4. Reinforces the importance of an Index as the Index now becomes the canonical way of even understanding the system.

What does everyone think?

I personally stumbled on this idea when trying to use Syncthing to move folders across devices. I’m a bit annoyed having to type long file-paths and worrying if I hit a filesystem file-path length name every time I move files around. Z:\11.21 is automatically the shortest possible JD-compatible file path, so why not use it exactly as is?

I’ve thought about this a lot too. In fact I started out with categories as the top-most level when I was just starting out.

A few thoughts. I’m definitely with you on exploring this seriously, not criticising at all.

browsability and muscle memory may be most important for a single user, but you could also use a JD system in a multi-user setting. There, the discoverability becomes important; the workbook has a lot of anecdotes relating to this. For a newcomer to the system, being presented with up to one hundred folders in a flat list would be overwhelming; being presented with ten (well-named) folders to start with allows an easier initial choice to be made. This is a point made in the workbook too.

I think if you have a full-text, fuzzy search interface (like most wiki/note apps do), you don’t need the areas in day to day use. I do have the feeling that areas are important in the design phase; although there, too, perhaps that might not be the case.

One idea I had was to apply the areas as an attribute of the category folders instead of as a level in the hierarchy. For example: the tagging software tmsu creates a virtual filesystem with directories for tags and symbolic links inside them to the tagged files. If directories 10 through 19 were tagged with 10-19 Area 1, then you could see that grouping in the virtual filesystem if you wanted. The ‘underlying’ layout would be flat, which would then facilitate the syncing and interoperability issues you mention (thanks for that; I hadn’t considered those aspects yet).

What we need is a file manager/interface which can be configured to hide a certain level in the file tree …

I do that to an extent by configuring my ‘fuzzy find’ (fzf) to use the command find with -mindepth 2 which then only matches level 2 and hides the empty area directories from my results list.

BUT, this is all platform-specific. The 10 - 10 - 100 directory tree is the most univeral format which satisfies the ‘soft’ objectives of the JD system.

Very interested to hear further thoughts on this topic!

@clappingcactus

I run (some) Linux and Windows, but since work-world is Windows - I’m mainly focused there.
(I have no clue about Apple or iOS.)

Don’t know if it tickles your fancy, but I run Everything as a countermeasure for “excessive clickyness” with folders and navigation.

Settings:
Under “Tools” > “Options” > “Indexes” > NTFS" I have disabled it from looking on my system drive and other areas.

Under “Search” > “Add to filters” I’ve added the path to where my JD structure is - and made this filter always loaded under “Settings” > “General” > “Home”.

In other words; It ONLY looks in my JD structure.

Everything is blazing fast, and I just have to throw some letters in there to find / discriminate down to the folder I want.

Always at the ready, a click a way - in the taskbar.

2024-07-09_19-08-33

Sidenote
I’ve seen “behind the scenes” of large, commercial operators that use file servers/shared network drives as their main storage and production systems. Yeah, they do have some internal rules and guidelines about how the folder structures should be - but since humans work there, it is not always so “clean” as it was intended. But they run Everything as a centralized resource from the IT department, where the filters and settings are done by the admins. That means; everyone there (can) use Everything to search/find what they are looking for. It’s quite efficient.

Interesting. To answer the question posed by the subject: ‘because it does’. That’s just how it happened at the time. (JD’s design is very accidental.)

When I used it in my work email, heavily and rigorously, I didn’t use area folders. I went straight to category, exactly for the less clicks reason. It worked great.

If it works for you, do it! But I think for most ‘normal people’ (i.e. people who probably aren’t on a forum nerding out about a decimal-based organisation system), the neat hierarchy is best.

What I like about it is that, once you’re ‘in’ an area, you should be able to totally forget about the others; cast them entirely from your mind. They’re not relevant to the thing you’re looking for. So if you can’t even see them, that should help.

That was how I immediately assumed it should be done when I read about taming email in the Workbook – only the Index needs the full hierarchy, everywhere else all you need is the AC.ID be it in a folder name, as a tag in your email program, or whatever.

And then if your index is one file or is a collection of notes which you can search-to-open (fzf or Everything or whatever), you kind of don’t see the areas most of the time.

@clappingcactus I’m interested in the technical reasons you alluded to, sync and interop with other tools. I’ve only skimmed the posts you linked to, yet. I don’t suppose that would be something that a JD Spec would help solve?

By the way this is really really enticing. On my site I have all short paths redirected to the full path, so I can just link to /21.01/ or whatever and that will expand out when the user clicks. It’s really really nice to be able to do that.

I’ve been thinking about this more. I think there’s this interesting tension between the usefulness of the tree structure for organising, and the usefulness of the flat structure for working faster.

It helped me tweak my own workflow a bit. I thought I’d share since it illustrates well how the tools you use can make all the difference.

  1. opening a JD tree in ranger, a multi-panel file browser.

  2. execute the command :flat 2 to unnest the tree to two levels. This shows all my IDs in one list. This is a useful place to be – I can search, sort by last modified, etc, and get work done.

  3. Now say I want to focus on one area. I execute the command :filter PATTERN to limit the entries in view: for example, :filter 01\. shows only IDs in category 01 Dogs.

  4. The filter persists, I can browse the child directories, open files etc, until I run :filter with no arguments to reset it.

Here it is in one motion:
recording-2024-07-11_22.16.57

So that’s fairly niche, and there are things to improve (it would be nice to hide the reduntant parts of the folder paths in flat view, for instance), but I hope it makes clear that the proper tooling, you can get pretty much any view you want on a plain ol’ filesystem hierarchy.

And, of course, we could just as easily go the other way round: make everything flat on the filesystem (to your point about interoperability), and add hierarchy with tags, symbolic links, virtual groupings using our tooling. This is really appealing to me – but I worry that I’ll end up with something like how my Zotero collection is at the moment, where I have all kinds of tags and categories I’ve forgotten about, and just resort to the big list of everything even though it’s sometimes slow. This is what the JD nested structure helps avoid.

thanks Fellow Programmer @SwissArmyWrench for the dummydecimal folder tree!

This is awesome! I’ve never tried Ranger, but the siren’s song of the command line is leading me to move more and more of my workflows into the terminal, so I might just have to give it a try!

Good to know that the DummyDecimal tree is being helpful. I created it for Johnnybgoode because I realized it A) helps privacy and B) would reduce a lot of confusion if multiple people were working on it and trying to resolve problems.

Ranger looks amazing! Added to the database of useful stuff at 31.02.

I’m going to play devil’s advocate for a few points in the thread to make sure all thoughts are on the table. I’m not particularly wed to my thoughts, and I myself have not re-done my file system based on this thinking. Just shooting ideas around.

Two points to respond to here:

  • Yes I think that discoverability is less important for individual users and more important in multi-user settings. For sure. But my take is that the biggest advantage of the system is to create a coherent address or location, not a path. If JD was a system to create the most sensible path to something we can just reduce it down to using some sort of alphabetical or chronological listing of files, with a pseudo-tag filetree that way every file is reachable from every location in a few clicks. This is obviously an extreme point to make, but I want to highlight that I think the strength of JD is in clarity about the system, and the ability to use the index for decision making, not the file-system. See the post below about using Everything to find things. You anyway wouldn’t want 20 users spending time browsing files as that adds up to man hours over years. You’d just come up with a fzf or file-naming scheme that’s standard across use-cases.

  • Optimizing the system for newcomers doesn’t have to happen at the file-system level, and should not saddle the system with a permanent design decision that was most helpful at the beginning of the organizational steps. I believe that newcomers also struggle with “too many areas” or “too many categories” syndrome. See: problems like the freelancer, or academic. It’s possible that doing away with a file-tree structure allows us to say “take all folders out of their current locations, no nesting allowed when beginning the process, now count your folders, now create the broadest possible folder category that encompasses most folders in your system and call that category 10-19.” This kind of education eases ramping on, naturally forces the maintainer to think as broadly as possible, and can hierarchially organize a user’s personal interests according to historical track-record. This is not what I suggest, but is an off-the-cuff example of the kind of JD development possible if we stop thinking in a file-tree first frame of reference.

Woah, I didn’t know that Everything can be limited in its search index. That’s awesome. But I think my point wasn’t necessarily about reducing clicks. It’s just more “what would the JD system and method look like if we didn’t actually put folders in folders”. That kind of idea would change how the system is designed, built, maintained, taught, used etc. Yes “Everything” the software absolutely solves the “used” part of my previous sentence. But I’m wondering what advantage JD would give to the average person, if they focused on “file storage clarification” rather than “folder storage design”.

That’s something I hadn’t considered. It of course helps to be “zeroed-in” on a topic. But that might still be equally effective working from ones index or notes rather than ones file-system right?

I think this begs the question of people in this thread: how many times do any of us perform a file-search-first action in our day to day lives? I find that for me as an academic, the answer is not much. Files are an endpoint, kind of like a game-save-state situation. If I need to cross-reference them, I already know where to look thanks to JD helping me organize at the outset. If I don’t know where to look, just like in the physical world, I’d check my notes first. But that comes from me being immersed in a note-first professional world.

So, it’s really boring. I stumbled on this idea because I use Syncthing to synchronize a set of JD areas. For me 10-19 is knowledge, 20-29 is projects, 30-39 is physical spaces. These three areas are always synchronized across all my devices. I then created a new ID elsewhere in the system, specifically 46.19 for “portable software” that’s supposed to live on a USB-stick. I realized I would want 46.19 on my laptop since there’s no point duplicating software if I have a portable version of it fulfilling the same function. To synchronize 46.19 I would then have to set up a shared folder in Syncthing from my NAS for all for 40-49 and include only 46.19, or I would have to synchronize 46.19 directly to my laptop and there create two folder hierarchies just to contain one new folder. Either way, there’s an unnecessary organizational tax and mess done on either the NAS side or laptop side.

Then it dawned on me that the same logic applies to all my files and folders. If I just had a flat folder ID structure, then the problem disappears, and on every device I only synchronize the IDs that that device needs, with exclusions and inclusions being handled at the syncing level. Not at the folder creation level.

It’s as if I’m dealing with my notes, I don’t really create a bunch of unnecessary folders to house a single note. I just write the note, reference it across the day, and if it’s relevant to a topic I make note in that specific topic that a new relevant note has been added to my notebook about it. Why handle files differently?

Absolutely true for me, too. I’m a mess myself. Just using JD to be less of a mess. I’m against tags (another topic). Except … that’s secretly what JD is right? Like Karl Voit would tell us all to “create a list of finite tags and apply as few as possible for each file”. And a JD system in effect forces the maintainer to work with a list of finite tags (at most 100) and assign as few as possible to each folder/project (1). It again comes back to the point that for me, someone who’s used my JD system since around late 2021, the biggest advantage has come from clarity about my system, and not from the folders. I have created knowledge of the files I like to store, and routines for how to store them and how to retrieve them. But now, at the point where I’ve memorized my IDs, it’s less important to remember with every single operation that 10-19 is knowledge.

Sorry for the long post!

1 Like

@clappingcactus , I for one am enjoying this discussion immensely. Great food for thought. My initial response was also kindof meant to play the devil’s advocate. I might try to keep arguing the other side – insofar as that exists here – to keep deepening the discussion.

However, this weekend is really full for me. So I’ll just mention a few things I can touch on briefly:

Intriguing distinction – to me – between the most coherent address versus most sensible path. I would think that the location is the path – that’s how it works at the neuronal level, as far as I understand it, and also how it works in an interlinked corpus with hyperlinks. Also, I would say the hierarchical folder system actually is the best way to make a sensible path to something. The guarantee about max two levels of depth, the use of decimal numbers instead of alphabetic sorting to ensure things stay in one place, for example. However:

Yes, yes, that resonates very much and sounds like a template for a way forward! I just think the original JD directory tree is a really good ‘poor man’s’ or ‘low tech’ index in itself, for people who don’t know how to set up some of the additional tooling layers that have been brought up.

I sometimes notice how, ironically, the process of thinking through were things belong has made their titles so memorable that I can just jump straight to them with a fuzzy search (without having to use the numerical ID[1]).

I use Syncthing too and I was planning on doing this at least at the category level. Maybe sync a few areas where I know beforehand if I know they will definitely belong on all devices. Have you noticed any performance issues if Syncthing is handling tens or hundreds of directories? Have you used the .stignore file at all? It works for keeping stuff off certain devices, but it is obviously not at all secure if you don’t trust those devices fully, and I thought there were a few cases where performance could be a problem.

Thanks for the Karl Voit mention! Great stuff!


  1. I try to always, after finding my target by title, erase my query and type the ID, just as a matter of discipline to get the numbers in my head … ↩︎

1 Like

@SwissArmyWrench @johnnydecimal ranger has definitely been growing on me. File and even image previews are awesome (a bit finicky to get set up) and after I’ve added a few custom commands (like integrating fzf) it gets better and better.

In fact, I modified the cool built-in :bulkrename command as a way to batch file things into their JD ID directories from the inbox.

  1. select a few files and run :jdmv – the list of files is opened in my text editor.
  2. add the AC.ID at the front of the filename – with autocomplete because this is my main editor with an fzf/find powered completion source :smiley:
  3. close the editor, and not only are the files renamed, but my script finds the corresponding directory and moves the files there.
1 Like