Web Directories: What Websites Are Made Of
Intro
Long ago I thought websites were just what you see on the screen. A homepage, some links, maybe a login page.
But then I learned that websites are like buildings — with hallways, storage rooms, locked cabinets, and sometimes forgotten closets. You just don’t see them in the main tour.
That’s what directory discovery is about: finding those hidden rooms.
🗂️ What Is a Web Directory?
When a browser requests a web page, it’s really just asking for a file on a server.
For example:https://example.com/about.html
This is a file in a directory, like /about.html.
Sometimes, there are entire directories like /admin/, /config/, /uploads/, or /backup/ that aren’t linked from the homepage but still exist.
If the server isn’t locked down properly, you can just go there. No password. No pop-up. Just… oops, here’s everything.
🔍 How Discovery Works
Directory discovery (or directory brute-forcing) is about trying to guess or enumerate these paths. Tools like gobuster, dirb, and ffuf take a wordlist (like admin, login, backup, etc.) and go:
https://example.com/admin
https://example.com/backup
https://example.com/.git
If they get a response (like a 200 OK), bingo — that path exists.
It’s like knocking on a bunch of doors and seeing which one creaks open.
🧪 What I Tried
I tested gobuster like this:
gobuster dir -u https://example.com -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt
This sends a bunch of directory guesses at the site and shows me which ones exist.
I didn’t find anything wild, but even seeing a /test/ or /dev/ path show up felt like finding a secret passageway in a video game.
🧠 Why It Matters
Sometimes, these directories expose config files, logs, or old versions of the site.
Developers might leave behind test folders or admin panels they forgot to remove.
In bug bounty and pentesting, these low-hanging fruits are often the start of bigger finds.
And it’s all legal… as long as you have permission or are using demo/test sites.
💡 What to Watch For
If you’re curious:
Visit
https://example.com/robots.txt— it often lists paths that site owners don’t want indexed 👀Try tools like
gobuster,ffuf, ordirsearchon intentionally vulnerable sites (like DVWA or bWAPP)Look out for status codes:
200 = exists
403 = exists but forbidden
404 = nope
🔁 TL;DR
Websites are more than what you see — there’s a whole folder structure underneath.
Directory discovery is about finding paths that weren’t meant to be public.
It’s beginner-friendly, surprisingly fun, and often part of the recon phase in ethical hacking.
